2026 email triage benchmark: GPT-4o vs DeBERTa 95th percentile latency vs recall

Premium Deals
Mighty Travels Premium
Travel in style,
save up to 90%

On flights and hotels worldwide by booking the best deals when they appear.

See Deals

Sponsored

TakeawayDetail
DeBERTa wins p95 latency and cost per decisionOn 2026 email triage, the DeBERTa-family encoder beats GPT-4o on 95th-percentile end-to-end latency and cost per decision, while GPT-4o keeps recall on threaded and ambiguous mail.
GPT-4o keeps recall on threaded and ambiguous mailThe encoder's recall loss is concentrated in threaded and ambiguous mail, so the split point must use per-class recall rather than aggregate recall.
Gate release on two same-host, same-holdout numbersMeasure p95 end-to-end latency and recall on the needs-action class at a fixed precision floor on the same holdout and same host.
Route only the classifier's low-confidence casesCompute the split point from p95 latency and per-class recall, with latency averaged over 20 runs per tool per day on one host.

On 2026 email triage, DeBERTa beats GPT-4o on p95 end-to-end latency and cost per decision, while GPT-4o keeps higher recall on threaded and ambiguous mail.

The routed-pipeline answer holds only when p95 latency and needs-action recall at a fixed precision floor are measured on the same holdout and host.

2026 email triage benchmark

How p95 email triage latency is clocked

The latency clock for this triage system starts at SMTP accept or webhook receipt and stops when the mail API acknowledges the label write; encoder inference is only one segment of that interval, so each message needs both timestamps logged rather than a model-call timer alone. The per-class recall denominator is the set of holdout messages whose ground-truth label is needs-action, measured at the fixed precision floor, so recall is true positives divided by that denominator rather than by all messages or predicted positives.

For the DeBERTa path, the measured work is tokenize subject plus first 256 body tokens, one forward pass, and argmax over labels; the series must include cold starts, retries, and queue waits alongside warm calls, because excluding them makes the p95 a fiction. If a retry changes the final label, the retried attempt still belongs in the same message's end-to-end interval.

For the GPT-4o path, clock prompt assembly, time-to-first-token, completion, and any retry or backoff through the same label-commit acknowledgement; do not report model latency as if prompt construction and API round trips were free. Both paths should be run on the same host and holdout so the p95 comparison is not confounded by infrastructure differences.

Report p95 over the full end-to-end series, not over a warmed subset, and pair it with needs-action recall at the precision floor from the same run; the release gate built on these two numbers is defined in the comparison table below, so this section only fixes how each number is clocked and scored.

Log, per message, receive timestamp, enqueue timestamp, model start, model end, label-write request, and API acknowledgement; the gap between receive and acknowledgement is the latency the user experiences, while intermediate stamps isolate queueing from inference. Reconcile missing acknowledgements as failures in the latency series rather than dropping them, because a dropped label has no completed end time.

How p95 email triage latency is clocked — 2026 email triage benchmark

What published benchmark methods settle

The convergence finding is that independent benchmarks do not compare systems on anecdotes: they measure repeated identical tasks from one fixed host and report distribution statistics, so a cross-model claim survives only when the task set, host, and repeat count are held constant.

BenchmarkFixed elementRepeated elementReported statistic
Hugging Face ai-coding-assistants-benchmark-2026Frankfurt VPS, 1 Gbps uplink20 runs per tool per dayFirst-token latency average
Assistant BenchmarkSame 15 tasks27 of 132 assistants tested1-10 real-use score
UBOS OpenClaw guidePredefined workloadRepeatable runsLatency, throughput, cost, accuracy, resource use

Hugging Face's ai-coding-assistants-benchmark-2026 measures first-token latency from a Frankfurt VPS with a 1 Gbps uplink, averaged over 20 runs per tool per day. Copy that fixed-host, N-repeats structure, not its latency numbers, which belong to coding assistants rather than mail triage; the check is whether your own host, network path, and repeat count are written down before the first run.

Assistant Benchmark scored 27 of 132 assistants on the same 15 tasks with a 1-to-10 real-use score. The transferable lesson is the identical task set: if the two triage paths do not face the same messages, the comparison is a story rather than a measurement, and a reader cannot tell whether the model or the sample changed.

UBOS frames benchmarking as a repeatable, quantitative process that evaluates an agent against a predefined workload, measuring latency, throughput, cost per inference, accuracy, and resource use. That is the same discipline as the fixed-host repeat runs above: define the workload once, rerun it, and compare distributions rather than single observations; a one-off run is a demo, not a benchmark.

What published benchmark methods settle — 2026 email triage benchmark

DeBERTa vs GPT-4o: the comparison table

The comparison is a measurement sheet, not a reputation score: every cell is filled from the same holdout and host, and the last column states how the number is checked rather than who claims it. Fill it by measurement, not reputation—no vendor claim, benchmark anecdote, or model name substitutes for a number produced on your mail, your host, and your labels.

Decision metricDeBERTa-class encoderGPT-4oHow you check
p95 end-to-end latencyLowerHigherMeasure the latency distribution on the holdout and report the 95th percentile
needs-action recall at precision floorLower on threaded and ambiguous mailHigher on threaded and ambiguous mailFix the precision floor, then score recall on the needs-action class
cost per 1,000 emailsLowerHigherMeasure total run cost against emails triaged on the same workload
low-confidence rateDefines the residualNot the routing triggerCount decisions below the confidence threshold divided by total decisions

Split routing wins this table: the DeBERTa-class encoder decides the high-confidence majority, and GPT-4o re-decides only the residual below the confidence threshold. Classifier-only wins latency and cost, GPT-4o-only wins ambiguous-thread recall, and neither wins both.

The low-confidence residual rule is the split point itself: route only the classifier's low-confidence decisions to GPT-4o, and set the threshold from the p95 latency budget and per-class recall rather than from convenience. A low-confidence decision is one whose score falls below the threshold; everything at or above it stays with the encoder. The threshold is not a tuning knob for accuracy alone—it is the point where the residual's added latency and cost still leave the whole pipeline inside the p95 budget and above the recall floor.

Tie-break rule: if more than one route satisfies both gate numbers, choose the one with the lower measured cost per 1,000 emails; if no route satisfies both, hold the release. This prevents a tie from being settled by model preference: the cheaper route that still clears both gates wins, and a failed gate stops the release regardless of the other metric.

Cost per 1,000 emails, not per token

The cost unit for this decision is cost per 1,000 triaged emails, not cost per token. A per-token figure hides thread length, output verbosity, and retry behavior; the per-1,000-email figure forces every model and route to carry the same denominator: one thousand messages moved from receipt to a committed label. Include encoder GPU time, API tokens, labeling, and escalation review in that numerator.

Compute the token portion as (input tokens + output tokens) multiplied by the current per-million-token price taken from the provider's live pricing page on the benchmark day, then add amortized GPU hours for the encoder. Do not reuse a prior-year price in a 2026 comparison; record the retrieval date and page with the result so a price change is visible rather than silent.

Charge the holdout itself as a cost line. Labeling the fixed evaluation set with two annotators plus an adjudication pass is a one-time expense, and it belongs in the benchmark budget. If it is treated as free research overhead, a low-token model can look cheaper than the routed pipeline it is being compared against.

Escalation review can dominate at scale. The low-confidence residual that leaves the classifier for human review carries labor cost that scales with escalation rate and reviewer time, not with token price. Measure the escalation rate on the same holdout, convert it to reviewer minutes per 1,000 emails, and include that labor in the same unit before comparing routes.

Use one accounting boundary for every candidate: same holdout, same price retrieval date, and same escalation policy. A table of cost lines keeps the comparison auditable.

Cost lineIncluded in per-1,000-email unit
API input and output tokensYes, at live per-million-token price
Encoder GPU hoursYes, amortized
Holdout labeling and adjudicationYes, as benchmark cost line
Escalation review laborYes, from measured escalation rate

Route on the fully loaded number: a candidate wins only if its cost per 1,000 triaged emails is lower while the needs-action recall floor is still met. If escalation review pushes the routed pipeline above the alternative, move the split point or change the residual rule rather than quoting token price alone.

Where the p95-vs-recall rule breaks

The p95-versus-recall rule is reliable only where the encoder sees the text that carries the decision. A 256-token window truncates quoted history in long threads, and that quoted chain often contains the request, deadline, or prior decision, so truncation removes signal rather than merely shortening input. Single-message triage can stay on the encoder while threaded mail goes to the GPT-4o path or a summarized context window.

Attachments and multilingual mail do not degrade both models in the same way. Stratify the holdout by stratum, report p95 end-to-end latency and needs-action recall separately for each stratum, and mark any stratum with fewer than 30 examples as unmeasured rather than as a pass. A stratum can pass on latency while failing recall, or pass on recall while adding tail latency, and aggregate numbers hide that tradeoff.

Break conditionRequired handling
Long threadsRoute to GPT-4o or use a summarized context window
Attachments or multilingual mailReport p95 and recall per stratum; unmeasured below 30 examples
Post-policy distribution shiftRecompute the split point on a fresh holdout

Post-policy distribution shift can invalidate the split point even when neither model changes: new senders, routing rules, or label definitions move the confidence distribution. Treat policy changes as a new measurement event, not as a configuration tweak, and recompute the split point on a fresh holdout before promotion.

Keep the gate paired at the stratum level. An aggregate p95 win does not excuse a needs-action recall loss in a named stratum, and an unmeasured stratum should be routed conservatively to the GPT-4o path until it reaches 30 examples and has both gate numbers.

Do not promote a release on aggregate metrics alone. Require the two gate numbers for every measured stratum, record unmeasured strata as explicit routing exceptions, and keep the release decision tied to strata that were actually measured.

500 emails, 4 checkpoints

Fix the evaluation inputs before the first request runs: draw a 500-email holdout from real traffic, confirm that 120 of those messages are labeled needs-action, and execute every measurement on one host in one region. Send 20 repeats per email so each model is exercised under the same load profile, and pin the model version and run date in the log header. These controls keep the two release numbers comparable on the same holdout and the same host, which is the gate this run is designed to satisfy.

Checkpoint 1 (p95 latency): for each model separately, collect one end-to-end latency per email, sort the 500 values in ascending order, and take the 475th value as p95. The index is fixed by arithmetic: ceil(0.95 × 500) = ceil(475.0) = 475. Never pool the two models into one distribution; a pooled percentile hides which system misses the tail-latency target and makes the release decision uninterpretable.

Checkpoint 2 (needs-action recall): compute recall as TP / (TP + FN) over exactly those 120 needs-action emails, and report it at the precision floor you locked before logging began. The denominator is the full labeled needs-action set, not the subset a model chose to score confidently and not the 500-email holdout total. If a model returns 96 true positives and 24 false negatives, recall is 96 / (96 + 24) = 96 / 120 = 0.80; changing the denominator after the run invalidates the gate.

Checkpoint 3 (false-negative cost): account for every missed needs-action message as downstream recovery work. A false negative leaves required action undiscovered until a human finds it later, so its cost is the operational burden of late detection, delayed handling, and any follow-up needed to repair the missed obligation. Record the count of false negatives from the same 120-email denominator so the team can weigh recall shortfalls against the latency and spend advantages observed in Checkpoint 1.

Checkpoint 4 (false-positive cost): account for every non-action message flagged as needs-action as avoidable interruption. A false positive consumes reviewer attention, displaces genuinely urgent mail, and adds unnecessary handling steps. Because the precision floor is fixed in advance, any gain in recall that drives precision below that floor converts directly into this error cost. The release is defensible only when p95 latency and needs-action recall pass together on this run and the error counts from both directions remain acceptable for the routed operating point.

Worked Example: Run the Numbers

Step 1, p95: using the end-to-end interval already defined for this system, let E be the encoder p95 and R be the residual model p95, with R > E. The residual share is 40 / 1,000 = 0.04, which is less than 0.05, so fewer than 50 of the 1,000 observations come from the slower route. Therefore the 95th percentile is not automatically the residual model's p95; it is determined by the encoder distribution plus at most the residual tail. Illustration: if E = 0.4 s and R = 2.0 s, the routed run's p95 stays near 0.4 s rather than becoming 2.0 s.

Step 2, recall: using the per-class recall denominator already defined for this system, overall recall is the message-weighted average of route-specific recalls. Illustration: if the encoder achieves 0.98 recall on the 960 routine messages and the residual model achieves 0.95 recall on the 40 routed messages, overall recall is (960 × 0.98 + 40 × 0.95) / 1,000 = (940.8 + 38.0) / 1,000 = 978.8 / 1,000 = 0.9788.

Illustration itemValue
Routine messages960
Residual messages40
Overall needs-action recall0.9788
p95 binding constraintencoder, below 5% residual share

For this illustration, the routed pipeline is the winner: it keeps the p95 near the encoder's value while producing 0.9788 needs-action recall, and it avoids sending all 1,000 messages through the slower residual model. The release gate still requires the residual model's own p95 and cost per decision to be acceptable, because those 40 messages are on the critical path.

The break-even trigger is a 5% residual share. Below 5%, the encoder's p95 remains the binding latency constraint; at or above 5%, the residual model's p95 becomes the binding constraint and the design must be rechecked against the gate. The recall break-even is the point at which the residual model's incremental recall on routed messages no longer exceeds the recall the encoder would have lost on those same messages.

Ship rules: five if/then gates

GateIfThen
1Classifier p95 is inside the SLO and needs-action recall clears the floor at locked precision.Ship classifier-only; reserve GPT-4o for audits.
2p95 misses the SLO while recall clears the floor.Cut the body window, batch requests, or add a queue; re-measure p95.
3Recall misses the floor while p95 passes.Do not ship classifier-only; test routing the classifier's low-confidence residual.
4GPT-4o beats classifier recall by at least 5 points on that residual and its p95 applies only to that residual.Ship split routing; monitor residual share weekly.
5Both models miss the latency SLO.Fix the pipeline before buying a larger model.

Gate 1 is the simplest release: the classifier is fast enough and accurate enough on needs-action mail, so GPT-4o stays out of the hot path and is used for audits. Gate 2 keeps the same model but changes the pipeline: if recall is acceptable and latency is not, cut the body window, batch requests, or add a queue, then re-run the same p95 measurement.

Gate 3 prevents a recall miss from being hidden by a passing latency number. If needs-action recall is below the floor, classifier-only shipping stops even when p95 is clean; the next test is whether routing the classifier's low-confidence residual changes recall without breaking the latency budget.

Gate 4 is the only topology change. It requires two conditions together: GPT-4o must beat classifier recall by at least 5 points on the low-confidence residual, and its p95 must apply only to that residual. If both hold, ship split routing and monitor residual share weekly; if either fails, stay on the current path. A rising residual share is a signal to re-run Gate 4, not to widen routing by default.

Gate 5 covers the case where neither model meets the latency SLO. Buy no larger model until the body window, batching, or queueing has been tried, because latency is an end-to-end property and a model swap does not remove queueing or payload cost.

Every gate is stateful: a change to the model, prompt, routing rule, host, or holdout resets the sequence, and the two release numbers must be re-measured before any ship decision.

What to do next

StepActionWhy it matters
1On the benchmark table above, compare the DeBERTa-family encoder and GPT-4o using p95 end-to-end latency and cost per decision, not mean latency or pooled F1.The release gate is defined by p95 latency, so averages or pooled F1 cannot override it.
2Inspect needs-action recall separately for threaded and ambiguous mail rather than relying on aggregate recall.The encoder's recall loss is concentrated there, so the split point must use per-class recall.
3Before release, confirm p95 end-to-end latency and needs-action recall at the fixed precision floor were measured on the same holdout and same host.The gate requires both numbers from the same holdout and same host.
4Set the routing split point from p95 latency and per-class recall, sending only the classifier's low-confidence residual to GPT-4o.This preserves GPT-4o recall where the encoder loses it without routing all mail through GPT-4o.
5Re-measure latency with 20 runs per tool per day on one host before changing the route.The split point depends on stable p95 latency, not a single measurement.
6Keep GPT-4o in the recall check for threaded and ambiguous mail after any encoder update.Prevents a silent recall regression in the residual route.

Frequently Asked Questions

Where is the DeBERTa encoder's recall loss most concentrated?

The DeBERTa-family encoder's recall loss is concentrated in threaded and ambiguous mail.

Why must per-class recall be used instead of aggregate recall when determining the split point?

Per-class recall must be used rather than aggregate recall because the encoder's recall loss is concentrated in threaded and ambiguous mail.

Which specific recall class must be measured to gate a release?

Release must be gated on measuring recall on the needs-action class at a fixed precision floor.

Under what conditions does the routed-pipeline answer hold?

The routed-pipeline answer holds only when p95 end-to-end latency and needs-action recall at a fixed precision floor are measured on the same holdout and same host.

Which cases should be routed from the classifier to GPT-4o?

Only the classifier's low-confidence cases should be routed to GPT-4o.

When does the end-to-end latency clock start and stop for this triage system?

The latency clock starts at SMTP accept or webhook receipt and stops when the mail API acknowledges the label write.

Quick answers

Which model wins on 95th-percentile end-to-end latency and cost per decision in the email triage benchmark?The DeBERTa-family encoder beats GPT-4o on 95th-percentile end-to-end latency and cost per decision.
On what types of mail does GPT-4o maintain higher recall?GPT-4o keeps higher recall on threaded and ambiguous mail.
What measurements are required to gate release?Release should be gated on p95 end-to-end latency and recall on the needs-action class at a fixed precision floor on the same holdout and same host.
How is the split point for routing determined?The split point is computed from p95 latency and per-class recall, with latency averaged over 20 runs per tool per day on one host.
Which cases should be routed in the pipeline?Only the classifier's low-confidence cases should be routed.

Also worth reading: Hand your travel logistics to an AI executive assistant: Hand your travel logistics to · Executive email style: Prompting vs Low-Rank Adaptation (LoRA) on 100-message holdout: Executive email style: Prompting vs · AI agent for performance review prep: grounded self-assessment vs recall — 2026 test: AI agent for performance review

Premium Deals
Mighty Travels Premium
Travel in style,
save up to 90%

On flights and hotels worldwide by booking the best deals when they appear.

See Deals

Sponsored

Research Methodology & Editorial Standards

We begin by defining the specific objectives the reader needs to accomplish. Primary product documentation and authoritative secondary sources are assembled into a verified research corpus; drafting occurs only after this foundation is in place.

Every quantitative claim is subjected to dual-source verification. Any figure that cannot be independently corroborated is either qualified or omitted.

Published · Last reviewed · Owned by the Withtai editorial desk (About, Contact, Privacy).

Related answers