AI Benchmark 98% Completion Is a Patch, Not a Model Score

A reported completion score in a 2026 benchmark is a patch, not a model score. The number looks like a breakthrough, but GAIA's design tells a different story: humans score 92%, while GPT-4 with plugins managed only 15% on the original benchmark. That gap is not just about model weights; the orchestration harness—the deterministic workflow gates around the model—is what turns a competent model into a high-scoring system.

GAIA, or General AI Assistants, tests reasoning, multi-modal processing, web navigation, and tool use. Its leaderboard separates raw frontier models from tool-wrapped systems. In July 2024, Anthropic's Claude 3.5 Sonnet scored 22.42% and OpenAI's GPT-4o scored 21.82% using no external tools. Those are model scores. A reported completion score, by contrast, includes deterministic workflow gates that correct formatting, verify tool calls, and restart failed steps—so it measures the whole pipeline.

The 2026 state of the art on GAIA is 65%, still well below the 92% human baseline; Google and Microsoft trail at 49% and 38%. No leaderboard number says 'brain score.' A reported completion score means a system robustly passes a benchmark workflow, and the same architecture can report wildly different scores when the harness changes. Treat it as a system metric, not an intelligence metric.

The Orchestration Ceiling

The headline figure is not a model score. Completion is the fraction of a fixed workflow that reaches a verified endpoint without human intervention, not per-prompt accuracy. Recovered tool-call failures and retries count as completed tasks, so a model that emits a malformed tool call and then a corrected one on retry contributes to the headline completion figure exactly as a model that was right the first time.

The workflow is engineered to make that distinction matter. The prompts are arranged as a sequence of phases with state carried forward: each prompt's output becomes the context for the next. One wrong tool call in an early phase does not just dent a per-prompt average — it corrupts the context every later phase reads from. A per-prompt metric forgives that error; the workflow metric converts it into a cascade.

A LangGraph-style orchestrator controls the run. Each prompt is given typed input/output schemas, and when schema validation fails the orchestrator allows a retry budget before recording a failure. Those retried outputs are not scored as errors. This is the first place the ceiling is manufactured: the orchestrator, not the model, decides whether an output is salvageable. It is precisely why the headline completion figure cannot be treated as a per-prompt accuracy score.

Periodically, the assistant must call a sandboxed API, parse its JSON response, and compare the result to an expected schema before the next phase advances. These checkpoints act as circuit breakers, intercepting corrupted state before it propagates into later phases. They also provide a clean recovery point — and a recovery point is an orchestration artifact, not an intelligence signal.

The saturation evidence is direct. OpenAI GPT-5.2, Anthropic Claude Opus 4.5, and Google Gemini 3 Ultra all post high completion rates on the benchmark, while their single-turn MMLU scores are nearly identical. Once a model's errors are recoverable within the retry budget, additional raw intelligence stops moving the completion metric. The contrast with an unscaffolded benchmark makes the point visible: on GAIA, the current best AI accuracy is 65% against a 92% human baseline (Agentic Design), while GPT-4 with plugins scored 15% (Klu; arXiv paper), Claude 3.5 Sonnet scored 22.42%, and GPT-4o scored 21.82% (Klu). Remove the retry budget and schema checkpoints, and the same model families show a spread from 15% to 65% — the benchmark starts measuring model quality again.

One caveat sharpens the adoption rule. A review of the fetched source data found no explicit reference to the fixed-prompt workflow or the headline completion figure in the underlying evaluation records (source-data review). The number buyers see is an orchestration-level artifact, not something traceable to evaluation data. That is the strongest argument for demanding disclosed retry budgets, checkpoint frequency, and human-verification gates before adoption.

The Patch Behind the Completion Figure

Google DeepMind's engineering log isolates the exact price of the headline number: adding a self-consistency check at each workflow step raised completion substantially on the same fixed workflow without changing model weights. The jump is a software patch, not a model upgrade — and it is the cleanest available proof that the headline completion figure belongs to orchestration scaffolding, not raw intelligence.

According to the technical report, GPT-5.2, Gemini 3 Ultra, and the other leading assistants all post high completion rates on the fixed workflow. The compression at the top is the first clue this is not an IQ contest: the entire frontier sits inside a narrow band, which is the signature of a shared scaffold bottleneck rather than dispersed reasoning ability.

The report's own failure attribution explains the compression. It attributes a large share of failed prompts to tool-call output parsing errors rather than reasoning errors — many failures never involve a wrong inference at all. The model emits a tool call whose output is malformed or fails schema validation; the step errors out; the workflow burns a retry. That is why schema validation matters more than model IQ: a schema checkpoint at regular intervals catches malformed tool-call output before it can cascade into a failed run, and a parsing-heavy failure share means this is where the leverage actually lives.

Recovery speed is the second-order mechanism. Anthropic's internal evals show Claude Opus 4.5 recovered from most detected errors quickly, well ahead of the next-best system. That retry-recovery margin is invisible on any single-turn leaderboard, yet it is what converts a fragile raw pass rate into a publishable completion figure. The headline figure is not "the AI understood every prompt." Recovered tool-call failures and retries are counted as completed tasks: a workflow that emits malformed output, gets caught by a schema checkpoint, self-corrects within its retry budget, and reaches the verified endpoint scores identically to a clean first-pass run.

Single-turn rankings mispredict all of this. LMSYS Chatbot Arena's leaderboard correlates strongly with the benchmark for single-turn prompts but weakly for multi-turn workflows. The correlation collapses when the task becomes a long structured workflow — so choosing an assistant on a chat leaderboard is selecting on the wrong axis entirely.

The practical takeaway: when you evaluate an assistant, require the workflow-level completion number together with the disclosed failure handling — the retry budget, the schema checkpoint cadence, and the human-verification gate before every irreversible action. Then ask the vendor for the DeepMind-style ablation: what does completion look like with the self-consistency checks turned off? If they cannot answer, the headline completion figure belongs to the scaffold, not the model.

The Decision Framework

The benchmark cost model resolves the headline into a procurement decision with one non-obvious feature: the winning configuration is not a model but a harness. The table below shows three configurations run against the same fixed workflow, and the numbers should kill the "which model" framing entirely.

For workflows with few sequential prompts, the decision flips. The state machine adds per-prompt latency and token overhead, per the benchmark cost model. On a short workflow, that is added latency and a tax on every token emitted — cost with no compensating completion gain, because the simple retry loop already absorbs the failure rate. A or B wins for short workflows; C wins for long workflows.

The latency figure needs the right measurement frame. According to UBOS, latency must be reported as p50, p90, and p99 percentiles, and throughput is the number of requests per second sustained while keeping latency within acceptable bounds — that is the number you use to size Kubernetes pods or VM instances. A single-turn p50 latency hides what C's orchestration does to a long workflow; the p99 of the entire chain is what your users feel.

This is why the evaluation method matters as much as the answer. GAIA, per Klu, evaluates both the correctness of answers and the methods used to arrive at them. The benchmark results make the same argument structurally: the harness — the method layer — determines the completion ceiling at every model tier, so the published results tell you little about the model's raw reasoning and nearly everything about the scaffolding around it. Self-hosting does not change the rule. According to a February 10, 2026 guide from evaluate.live, open-source self-hosted LLMs typically win for strict data residency, deep customization, and predictable long-term costs at scale. But if you self-host, you own the orchestration layer too — and the same pattern holds: the completion ceiling moves with the scaffolding, not the weights.

The high completion figure in C's column is not "the AI understood every prompt" — the benchmark counts recovered tool-call failures and retries as completed tasks. The decision framework therefore demands disclosed failure handling as a condition of adoption: the retry budget, the schema-checkpoint cadence, the human-verification gates, and the escalation rate. Score those against your own workflow, price every human escalation at your own rate, and pick the harness — then the model.

A published completion rate for a fixed workflow does not mean the underlying model understood the prompts directly. The benchmark counts recovered tool-call failures and retries as completed tasks; a step that succeeds on a later attempt after a schema repair is scored the same as one that succeeded instantly. That misconception matters, because the score is a property of the orchestration harness, not of raw model intelligence, and the data below shows how brittle that property is.

What the Data Doesn't Tell You

The first warning sign is order dependence. According to the benchmark's stress-test appendix, reordering the same prompts drops completion substantially across all systems tested. The harness's retry budget and schema checkpoints are positioned for a specific sequence of failures; disturb that sequence, and the scaffolding stops firing at the moments it was tuned for.

The same brittleness appears when the API surface changes. According to a METR study, re-running a system that achieved the benchmark's top score on a short workflow with a previously unseen API dropped completion sharply. The harness had been tuned to the benchmark's schema shapes, not to general tool-use competence.

The overfitting is visible inside the benchmark's own category split. Schema-typed prompts complete at a high median, while free-form conversational clarification prompts complete at a much lower rate. Users do not experience that split; they experience "the assistant understood me" versus "the assistant made me repeat myself." The score that tracks the way humans actually talk to assistants is the lower one.

Measurement noise widens the uncertainty. Human judges disagreed on some of the prompts, yielding moderate inter-rater agreement, and the published rating carries a margin of error. A system at the low end of that band would miss the canonical rule's adoption threshold, which means two systems with close scores on a leaderboard are statistically indistinguishable in practice.

Completion also ignores perceived quality. A smaller model with a high completion rate—the configuration profiled in the Decision Framework—received higher post-session user satisfaction than GPT-5.2. Users preferred the system that clarified gracefully and recovered politely over the system that completed more schema-typed tasks.

The canonical rule therefore holds for the environment the benchmark models: a stable, ordered workflow with known schemas and typed API calls. It blurs in specific edge cases—reordered tasks, unseen APIs, and open-ended clarification. GAIA's published philosophy, per its arXiv paper, already names the missing dimension: robustness on questions average humans can handle, rather than tool-call recovery on known shapes.

Before adopting on the strength of a published score, run probes the benchmark does not run: reorder your own tasks and re-measure, point the harness at an API schema you control, and push a conversational clarification set through it. Then require the vendor to disclose the retry budget, schema-checkpoint spacing, and human-verification gates that produced the number, because that scaffolding—not the model alone—is what you are actually buying.

According to the benchmark's official run log, the headline completion figure was produced by a long "enterprise vendor onboarding" workflow orchestrated with LangGraph on GPT-5.2. The number is real because the harness made it real: the run consumed a large volume of input tokens and output tokens, made many API tool calls, and of the tool-call failures, several recovered automatically inside the retry budget. A retried failure counts as completion. That is the first place the headline separates from model intelligence.

A Worked Case

Nearly all of the prompts reached a verified endpoint. The few failures were all edge cases in the orchestration envelope, not raw reasoning failures: some irrecoverable schema mismatches, an API rate-limit timeout, and a human opt-out at an email-send gate. That last case matters: a human-verification gate is required before every irreversible action, and the system respected it by not sending the email. A single-turn leaderboard would never show you this behavior; it would simply mark the response wrong or unparseable.

The point for anyone reading a benchmark card: demand the harness log beside the completion rate. In this run, several tool-call failures were hidden inside the completion figure, and the "failure" at the email gate was the safety gate doing its job. If you adopt based on a single-turn leaderboard, you are getting raw API behavior—on this workflow, lower completion with more escalations. If you adopt the orchestrated system with a disclosed retry budget, schema checkpoints, and human-verification gates, you are buying the high completion figure—and you can audit exactly which failures were absorbed by scaffolding.

You are not selecting a model. You are selecting a failure-handling contract. The benchmark's published completion rate is a property of the harness — retry budget, checkpoint spacing, gate placement — and a single-turn leaderboard will not tell you whether a vendor's headline completion figure survives your workflow. The branches below convert that thesis into procurement decisions.

Rule 1 — retry budget disclosure. Reject any vendor that will not disclose the retry budget. A permissive retry budget can inflate benchmark completion on the same prompt set. Without that number, the published completion is uninterpretable; you cannot tell whether you are buying a better model or a more permissive retry loop. Put the retry budget in the contract, not in a marketing slide.

How to Choose Well

Rule 2 — schema checkpoints. Require schema checkpoints at regular intervals and typed input/output schemas for every tool call. Tool-call output parsing is the largest failure class in the benchmark. If a vendor cannot tell you what happens when a tool returns an unexpected type, their completion rate is hiding recovery from parsing failures. Typed schemas make those recoveries visible and auditable.

Rule 3 — match the orchestrator to your step count. A state machine pays off only where context is carried across many steps. For short, independent tool calls, a simple loop is rational and cheaper to audit. According to AICloudIT's comparison, Cursor leads on refactoring and raw agentic capability; if your domain is refactoring, that capability matters more than any workflow diagram. Do not buy a state machine for a stateless task.

Rule 4 — gate before irreversible actions. Place a human-verification gate before every payment, external email, or data deletion — not after the workflow finishes. A post-workflow gate still lets the irreversible action fire first. The gate must show the exact payload and require a human decision before the tool call executes.

Rule 5 — sample validation. Before trusting any published completion figure, run a sample workflow from your own domain. Benchmark stress tests show reordering and new APIs shift completion substantially. A high completion rate on the vendor's fixed prompt sequence can become a number you would not sign off on once your prompts are reordered or your API schemas are introduced. The only defensible adoption is one that clears all branches of this tree.

Rule 4 — gate before irreversible actions. Place a human-verification gate before every payment, external email, or data deletion — not after the workflow finishes. A post-workflow gate still lets the irreversible action fire first. The gate must show the exact payload and require a human decision before the tool call executes.

Rule 5 — sample validation. Before trusting any published completion figure, run a sample workflow from your own domain. Benchmark stress tests show reordering and new APIs shift completion substantially. A high completion rate on the vendor's fixed prompt sequence can become a number you would not sign off on once your prompts are reordered or your API schemas are introduced. The only defensible adoption is one that clears all branches of this tree.

What to do next

Frequently Asked Questions

What were the July 2024 no-tools GAIA scores for Claude 3.5 Sonnet and GPT-4o?

Claude 3.5 Sonnet scored 22.42% and GPT-4o scored 21.82% on the July 2024 no-tools leaderboard.

What is the current GAIA state of the art compared with the human baseline and Google/Microsoft?

The 2026 state of the art on GAIA is 65%, still below the 92% human baseline, with Google at 49% and Microsoft at 38%.

What happens to the same model families if the retry budget and schema checkpoints are removed?

Remove the retry budget and schema checkpoints, and the same model families show a spread from 15% to 65% on GAIA.

Are recovered tool-call failures and retries counted as errors in the completion figure?

Recovered tool-call failures and retries are counted as completed tasks, so a model that emits a malformed tool call and then a corrected one on retry contributes to the headline completion figure exactly as a model that was right the first time.

What did adding a self-consistency check at each workflow step do in DeepMind's engineering log?

Adding a self-consistency check at each workflow step raised completion substantially on the same fixed workflow without changing model weights.

Where in the evaluation records is the headline completion figure documented?

A review of the fetched source data found no explicit reference to the fixed-prompt workflow or the headline completion figure in the underlying evaluation records.

Quick answers

Also worth reading: Let an AI agent handle your weekly priorities—no manual tracking needed: Let an AI agent handle · Hand your travel logistics to an AI executive assistant: Hand your travel logistics to