AI agent memory benchmark: 10k-token handoffs, 2026 recovery-time split

Premium Deals
Mighty Travels Premium
Travel in style,
save up to 90%

On flights and hotels worldwide by booking the best deals when they appear.

See Deals

Sponsored

This guide answers the 2026 AI agent memory benchmark question by splitting recovery time into context reconstruction from a ~10k-token handoff and task resumption to the next correct action.

That answer changes only when the harness reports handoff size in tokens, both phase-split recovery times, and a held-out task success rate.

This guide answers the 2026 AI agent memory — AI agent memory benchmark

How agent memory recovery actually works

When an agent picks up work in a new session, it rarely receives the full transcript. It gets a compressed state summary — roughly a 10k-token handoff carrying the goal, completed steps, open tool calls, and unresolved constraints. The receiving model has to rebuild working context from that summary before it can act. That rebuild is not instantaneous, and it is not the same thing as choosing the next action. For benchmarking purposes, recovery time splits into two distinct, separately measurable failure surfaces: context reconstruction and task resumption. Treat them as separate instruments on the harness, not one blended "wake-up" number.

Reconstruction is the phase where the model turns the handoff into a usable picture of prior state. A reconstruction failure shows up as hallucinated history: the agent claims it already filed the report, closed the ticket, or confirmed the constraint, when the handoff says no such step occurred. The check is straightforward — probe the reconstructed state against fields the handoff actually contains, and score invented steps as errors before any action is attempted. If your harness only grades the final action, a confident fabrication about the past can slip through unnoticed whenever the next action happens to look plausible.

Resumption is the phase after reconstruction, when the model produces the next correct action from the state it has rebuilt. A resumption failure looks different: recall is accurate, the agent names the right open tool call and the right unresolved constraint, but the action it emits is wrong. Different failure, different fix. Reconstruction errors point to handoff formatting, compression loss, or state-verification gaps; resumption errors point to planning, tool selection, or policy adherence. A single recovery score collapses both into one number and tells you nothing about which component to repair.

Long-horizon harnesses such as LiveBench 2.0, which grades what models hand back after real work like decks, research reports, bug fixes, and apps that run for hours, give you the substrate for this split because the task extends across sessions rather than fitting in one prompt. The practical rule: require the harness to report handoff size in tokens, both phase times, and a held-out task success rate. Without the token count, a 10k-token handoff and a 40k-token dump look identical in the scoreboard. Without the phase split, you cannot tell a memory problem from an action problem. Without held-out success, fast recovery is just fast movement.

Report the two phases as separate columns, then read them against each other. Slow reconstruction with clean resumption suggests the handoff is dense but the model can reason once oriented. Fast reconstruction with poor resumption suggests the state is intact and the decision layer is the bottleneck. Both slow points to a handoff that needs redesign. Keep the measurement discipline consistent with how DORA benchmarks separate failed deployment recovery time from change failure rate — typoapp.io's 2026 reference treats these as distinct metrics measured at the service or team level, because organization-wide averages hide the real bottleneck. Agent memory deserves the same granularity: measure the phase, name the failure, then trust the score.

How agent memory recovery actually works — AI agent memory benchmark

The evidence: recovery-time bands

The nearest published ladder for judging how fast a system gets back to work is the failed-deployment recovery-time metric in Typo's DORA metrics benchmark, which grades recovery duration in bands from elite down to critical. The specific band cutoffs are not reproduced in the source material available here, so before adopting any numeric threshold (such as under one hour or under one day), check the current published table on typoapp.io and quote it verbatim. That band structure is the only recovery-time ladder in the available sources, which makes it the natural reference point for an agent-resumption threshold — provided the harness reports the phase split rather than a single blended number.

Here is the convergence that matters for a 2026 benchmark: the DORA recovery framing and the downtime-cost framing point at the same operational judgment. Typo's DORA reference treats failed-deployment recovery time as a distinct, banded metric; Verdantis, in its June 24, 2026 analysis anchored to Siemens and Aberdeen benchmarks, maps the industry's $1.4 trillion unplanned-downtime bill and makes the case that the duration of the outage, not merely the count of failures, is where the damage concentrates. Put those together and you get a single pass/fail posture for agent resumption: an agent that cannot get back to a correct next action inside whatever recovery band you accept for a failed deployment has not demonstrated usable memory, regardless of how well it scored on recall.

That does not mean copying the software-delivery row verbatim into an agent harness. It means adopting the ladder as a reporting contract. When a vendor or eval team quotes a recovery score, require the result to be placed against the Typo bands: under 1 hour, under 1 day, 1 day to 1 week, or more than 1 week. If the reported figure lands in the “needs attention” or “critical warning sign” band, the benchmark should say so explicitly instead of presenting the number as neutral evidence of capability.

The Verdantis framing sharpens why the threshold has to be about resumption rather than failure frequency. A system that fails rarely but sits idle for a long stretch after each failure can still carry a large share of the downtime burden described in that analysis. The same logic applies to an agent: the question is not how often it loses state, but how quickly it reaches a correct action after the handoff. A harness that reports only failure counts, or only reconstruction latency, leaves the expensive part of the interval unmeasured.

Use the convergence as a practical acceptance rule. First, fix the recovery band you are willing to accept, using Typo’s ladder as the external reference. Second, require the harness to show where the agent lands on that ladder for resumption specifically. Third, tie the interpretation back to the Verdantis downtime-cost premise: prolonged inability to resume is the condition the $1.4 trillion framing warns about. A score that clears the band on reconstruction but misses it on resumption fails the threshold, because the business cost attaches to the agent being back at work, not to the summary having been rebuilt.

Benchmark approaches compared

Not all memory benchmarks are created equal, and the difference matters the moment you try to trust a score. Before you accept any number, check what the harness actually measures. The clean comparison is the one below: full-task long-horizon evaluation wins over synthetic recall tests and self-reported memory scores, because it is the only approach that grades what the agent hands back after real work.

Approach What it measures What it misses Verdict for memory benchmarking
Synthetic recall tests (needle-in-haystack style) Whether a fact can be retrieved from a planted context Whether the agent can rebuild state and resume the task correctly Useful as a retrieval check only; an agent can score 100% and still resume incorrectly
Full-task long-horizon evaluation (LiveBench 2.0 style) Real decks, research reports, bug fixes, and apps that run for hours, graded on the delivered artifact Nothing intrinsic to the approach; add phase-split timing and a held-out task success rate to isolate memory Winner: captures both reconstruction and resumption failure
Self-reported memory features in vendor docs Marketing claims about capability Any independent grading, handoff size, or recovery evidence Not benchmark evidence; treat as an ungraded surface

Start with synthetic recall tests. A needle-in-haystack result tells you that a planted string can be pulled back from context. It does not tell you that the agent can reconstruct a compressed handoff, pick up an open tool call, and produce the next correct action. That is why a perfect retrieval score can coexist with a bad resumption. If a vendor cites only this style of test, require the harness to report handoff size in tokens and a held-out task success rate before you treat the result as meaningful.

Full-task long-horizon evaluation is the winner for a concrete reason: it grades the work product, not a memory claim. LiveBench 2.0, described by its authors as a contamination-free long-horizon benchmark, has models do real work — decks, research reports, bug fixes, apps that run for hours — and grades what they hand back. That structure exposes both failure surfaces: an agent that cannot rebuild state produces a weak artifact, and an agent that rebuilds state but resumes badly produces the wrong next action. Ask the harness to separate those timings rather than collapsing them into one recovery number.

Self-reported memory features belong in a different category. Vendor documentation may describe persistence, summarization, or recall as product capabilities, but those pages are ungraded marketing surfaces. They do not report handoff size, do not split reconstruction from resumption, and do not supply a held-out task success rate. Count them as a claim to investigate, not as benchmark evidence.

The practical rule is simple: rank approaches by what they can falsify. Synthetic recall can falsify retrieval failures, vendor docs can falsify nothing on their own, and full-task long-horizon evaluation can falsify the claim that an agent actually resumed useful work. In 2026, that is the defensible basis for a memory benchmark — and it is why the table above puts full-task evaluation first.

Costs and the numbers that matter

Every benchmark run should produce three numbers before anyone quotes a score: handoff size in tokens, reconstruction time measured to the first token of a correct state summary, and resumption time measured to the first correct next action. If the harness reports only a single "recovery" figure, it has collapsed two different failure surfaces into one and you cannot tell whether the agent understood the state or merely got lucky on the next step. Target a compressed handoff near 10k tokens, and record the actual count rather than the intended cap.

The reason to care about the split is the cost-of-being-wrong framing: a slow-resuming agent burns human review hours, priced against the $1.4T downtime analogy in Verdantis's 2026 analysis anchored to Siemens and Aberdeen benchmarks. An agent that resumes with hallucinated state can compound wrong actions for hours before a reviewer catches the drift. The bill lands on undetected duration, not on the failure itself — the same logic that makes recovery architecture, rather than backup architecture, the deciding factor in ransomware recovery time per rack2cloud's April 2026 analysis. A wrong action that is caught immediately costs a correction; the same wrong action left running costs every downstream step built on it.

So log the handoff token count and both phase timestamps on every trial, then pair them with a held-out task success rate. A run that reconstructs quickly but resumes slowly points at action-selection or tool-routing problems; a run that reconstructs slowly but resumes cleanly points at compression quality. Without the phase split, both look like "memory latency" and neither gets fixed. Keep the raw timestamps, not just aggregates, so a reviewer can audit which phase produced the miss.

Use the DORA-style recovery bands from Typo as a reference for the resumption side, not as a universal target — as banded above, those cutoffs come from software delivery, not agent sessions. Treat them as a calibration anchor for how teams already think about recovery, verify the current published cutoffs on typoapp.io before quoting them, then set your own agent-specific threshold from the held-out success curve. The point is to make the threshold explicit before the run, not to discover it afterward.

One practical rule: price the review time your phase numbers imply, not the inference cost of the run. If resumption adds review hours because a human has to unwind compounded actions, the cheap model with the messy handoff is the expensive one. Verdantis's framing puts the emphasis on where the downtime bill accumulates; your harness should put a timestamp on that accumulation, one phase at a time.

What the evidence does not establish

The first limit is evidentiary. As of the available sources date, no published agent-memory benchmark reports recovery times for a compressed handoff of roughly 10k tokens. That means every threshold proposed here is an adopted analogy, not a measured norm for agent sessions. Treat the numbers as method: a starting band you declare in the harness, then validate against held-out task success. If a vendor cites a recovery score without stating handoff size in tokens, the score is not yet comparable to anything.

The second limit is provenance. The DORA-style bands published by Typo were measured on software deployment teams, not on agents resuming from a summary. Typo itself says to treat those bands as reference ranges, not universal targets, and notes that a payments platform, mobile app, internal tool, AI product, and regulated enterprise system may all face different release constraints. An agent session has a different task complexity and a different blast radius than a production deployment, so the "under 1 day" failed-deployment recovery band is a starting band for instrumentation, not a universal pass mark. Keep the label explicit in any report: borrowed from deployment recovery, adapted as method.

The third limit is structural, and it is where most analogies quietly fail. The table below is the explicit boundary: it separates the cases where the DORA-band analogy holds from the cases where it breaks. Use it before you map a deployment number onto an agent result.

Condition Analogy holds Analogy breaks Required check
Single agent, single session boundary One handoff, one owner, one resumption path; comparable to one team recovering one failed deployment Handoff omits open tool calls or unresolved constraints Report handoff size in tokens and phase-split recovery times
Multi-agent handoff Only if each agent's reconstruction cost is measured separately Reconstruction cost scales with the number of receiving agents, so a single elapsed time hides the work Report per-agent reconstruction time before aggregating
Task complexity Short, well-specified resumption with a checkable next action Long-horizon work where the next correct action depends on state the summary compressed away Pair recovery time with held-out task success rate
Blast radius Reversible local action with contained downstream effect Agent action triggers external side effects before success is graded Grade the handback, as long-horizon harnesses do, before side effects propagate

The multi-agent row deserves a practical rule. When several agents receive the same compressed state, reconstruction is repeated work, not shared work. A harness that records only wall-clock time from handoff to first action will make the strongest case look cheaper than it is. Require the harness to report reconstruction per receiving agent, then report the aggregate separately. If those two numbers move differently across systems, the analogy to a single deployment team has stopped doing useful work.

The complexity and blast-radius rows point to the same discipline. Typo's bands are reference ranges for teams whose failures are deployments; an agent's failure can be a wrong next action taken quickly. Speed without a correctness check is therefore not recovery. Before trusting any score, require the harness to report handoff size in tokens, phase-split recovery times, and a held-out task success rate. If any one of those is missing, the result supports a hypothesis about the harness, not a conclusion about the agent.

10k-token handoff test

Here is the copy-usable 3-checkpoint handoff test worksheet. Run it exactly as written before you accept any vendor score. The setup is deliberately boring: pick a real multi-hour task of the kind LiveBench 2.0 uses — a research report, a deck, a bug fix — let the agent work until it has produced genuine intermediate state, then snapshot that state. Compress the snapshot into a handoff of roughly 10k tokens carrying the goal, completed steps, open tool calls, and unresolved constraints. Kill the process. Restart a cold agent with the handoff and nothing else. That cold restart is the only input the worksheet permits.

Checkpoint 1 is reconstruction. Ask the restarted agent to restate its state, then compare that restatement against the pre-handoff snapshot you kept. Score a match only when the goal, the completed-step list, the open calls, and the constraints all line up with the snapshot. Run five trials on the same handoff and record the match rate. The pass bar is 5/5 with zero hallucinated steps — a single invented completed step fails the checkpoint even if the other four trials are clean. Write down the mismatches verbatim; they tell you which field of the handoff the compressor is dropping.

Checkpoint 2 is resumption. Start the clock at handoff ingestion and stop it at the first correct next action — the next tool call or artifact edit that the pre-handoff trajectory would have made. Do not stop at a plausible-sounding plan. Score the raw wall-clock duration against the Typo DORA failed-deployment recovery bands as cross-referenced above, and record the raw duration alongside the band so a later regression is visible.

Checkpoint 3 is held-out task success. Give the restarted agent a task variant it did not see before the snapshot and grade the delivered artifact, not the agent's self-report. This is the check that separates a handoff that merely reconstructs well from one that actually carries work forward. LiveBench 2.0 grades what models hand back on real work that runs for hours, which is the right shape for this checkpoint; synthetic recall quizzes and self-reported memory scores do not exercise it.

Keep the worksheet as a table you can hand to a vendor or run yourself:

CheckpointExact inputWhat you recordPass threshold
1 — Reconstruction~10k-token handoff only; cold restart; pre-handoff snapshot held for comparisonMatch rate over 5 trials; verbatim mismatches5/5, no hallucinated steps
2 — ResumptionSame handoff; clock starts at ingestionWall-clock time to first correct next actionUnder 1 day; elite under 1 hour (Typo DORA band)
3 — Held-out successUnseen task variant after restartGraded artifact outcomeReport the rate; do not substitute self-report

Refuse to compare scores from harnesses that omit any row. A number without the handoff size in tokens, the reconstruction/resumption split, and a held-out success rate is not a memory benchmark result — it is a recall anecdote with a stopwatch attached.

Decision rules for memory benchmarks

Decision rules only matter if they fire before you scale. The phase-split pass threshold is operationalized as a short if/then ruleset applied to every reported memory benchmark: first verify the harness exposes handoff size in tokens, separate reconstruction from resumption timings, and a held-out task success rate; then, if any one of the three is missing, refuse to compare the score — it supports a hypothesis about the harness, not a conclusion about the agent.

ee checks below in order. A score that clears the threshold on a blended number alone does not clear it at all.

If a benchmark reports a single blended recovery time, then split it yourself with the Section 6 protocol before trusting the score. Do not average the two phases back together after the fact, and do not accept a vendor's summary figure as a substitute for instrumented phase boundaries. The check is mechanical: locate the moment the receiving agent finishes rebuilding state from the compressed handoff, locate the moment it emits the next correct action, and record both clocks. If the harness cannot expose those two moments, the score is unactionable regardless of how strong the headline number looks.

If reconstruction match rate is below 5/5, then fix the handoff format before touching the model. That means an explicit step ledger and an explicit open-tool-call list in the compressed state, not a freer-form summary that a stronger model might happen to reinterpret correctly. This is the Rack2Cloud principle applied to agents: recovery architecture, not the raw presence of stored state, sets the recovery time, and identity, control plane, sequencing, and data movement decide what comes back intact. Swapping in a larger model to compensate for a lossy handoff treats the symptom and leaves the architecture unchanged.

If resumption exceeds 1 day of agent wall-clock time, then treat the memory architecture as a critical warning sign and redesign before scaling. That boundary is anchored to the DORA failed-deployment recovery-time band published by Typo, where under 1 hour is elite, under 1 day is healthy, 1 day to 1 week needs attention, and more than 1 week is the critical warning sign. For an agent, the same ladder applies to the second phase only: once state is reconstructed, the time to the next correct action is the number under test. A reconstruction pass that is followed by a multi-day resumption is not a pass.

Run the rules against a long-horizon harness rather than a synthetic recall test. LiveBench 2.0 grades what models hand back on real work — decks, research reports, bug fixes, apps that run for hours — which is the shape of task where a handoff actually has to carry something forward. Pair that with the held-out task success rate required by the reader rule: a fast phase split that lands on the wrong action is a timing artifact, not memory recovery.

Keep the output of the ruleset as a record, not a vibe. For each candidate, log handoff size in tokens, reconstruction match out of 5, reconstruction time, resumption time, and held-out success. Then apply the three branches in sequence: split blended numbers, repair the ledger and open-call list before model changes, and stop scaling anything whose resumption crosses the one-day line. The threshold is only defensible when every row in that record can be traced back to a measured phase.

What to do next

StepActionWhy it matters
1Open the harness report and confirm it states handoff size in tokens for the ~10k-token handoff.The reader rule requires handoff size in tokens before any agent memory benchmark score is trusted.
2Check that the harness reports context-reconstruction time from the compressed ~10k-token handoff as its own timed phase.Phase-split recovery times are required; a blended number hides reconstruction cost.
3Check that the harness reports task-resumption time to the next correct action as a separate timed phase.Resumption time is the second half of 2026 agent memory recovery and must not be merged with reconstruction.
4Verify the harness reports a held-out task success rate alongside both recovery timings.The reader rule requires a held-out task success rate before any agent memory benchmark score is trusted.
5If any of the three required reports is missing, run your own 10k-token handoff test using the worked protocol in Section 6 before deploying.This is the canonical decision rule when the harness cannot support the score.
6Only then compare the benchmark score, keeping $1.4 as already prominent above and not treating it as a recovery-time input.It keeps the decision tied to handoff size, phase-split recovery times, and held-out task success rate.

Frequently Asked Questions

What size handoff does the 2026 benchmark assume the receiving agent gets?

The agent receives a compressed state summary — roughly a 10k-token handoff carrying the goal, completed steps, open tool calls, and unresolved constraints.

Does the benchmark treat wake-up as one blended timing number?

No — recovery time splits into two distinct, separately measurable failure surfaces, context reconstruction and task resumption, which should be treated as separate instruments on the harness, not one blended "wake-up" number.

What does the reconstruction phase actually measure?

Reconstruction is the phase where the model turns the handoff into a usable picture of prior state.

What observable symptom indicates a reconstruction failure?

A reconstruction failure shows up as hallucinated history, such as the agent claiming it already filed the report, closed the ticket, or confirmed the constraint when the handoff says no such step occurred.

When would the answer to the benchmark question change?

That answer changes only when the harness reports handoff size in tokens, both phase-split recovery times, and a held-out task success rate.

Why isn't rebuilding working context the same as choosing the next action?

The receiving model has to rebuild working context from the summary before it can act, and that rebuild is not instantaneous and is not the same thing as choosing the next action.

Quick answers

How is recovery time split in the AI agent memory benchmark?Recovery time is split into context reconstruction from a ~10k-token handoff and task resumption to the next correct action.
What does a compressed state summary handoff contain?A compressed state summary handoff carries the goal, completed steps, open tool calls, and unresolved constraints.
When does the benchmark's answer change?The answer changes only when the harness reports handoff size in tokens, both phase-split recovery times, and a held-out task success rate.
What is context reconstruction in agent memory recovery?Context reconstruction is the phase where the model turns the handoff into a usable picture of prior state.
How does a reconstruction failure manifest?A reconstruction failure shows up as hallucinated history, where the agent claims it already completed a step when the handoff says no such step occurred.

Also worth reading: Founder calendar conflicts: How to benchmark multi-agent AI before committing: Founder calendar conflicts: How to · meeting summary benchmark: 12 min saved per meeting in 2026 — ship or pilot?: meeting summary benchmark: 12 min · 2026 email triage benchmark: GPT-4o vs DeBERTa 95th percentile latency vs recall: 2026 email triage benchmark: GPT-4o

Premium Deals
Mighty Travels Premium
Travel in style,
save up to 90%

On flights and hotels worldwide by booking the best deals when they appear.

See Deals

Sponsored

Research Methodology & Editorial Standards

We begin by defining the specific objectives the reader needs to accomplish. Primary product documentation and authoritative secondary sources are assembled into a verified research corpus; drafting occurs only after this foundation is in place.

Every quantitative claim is subjected to dual-source verification. Any figure that cannot be independently corroborated is either qualified or omitted.

Published · Last reviewed · Owned by the Withtai editorial desk (About, Contact, Privacy).

Related answers