| Takeaway | Detail |
|---|---|
| F1 winner flips by transcript length | GPT-4o and Claude 3.5 trade action-item extraction leadership across 300 call transcripts depending on call duration. |
| Stratify the corpus into three length buckets | Segment the 300-call corpus into under 15 min, 15–45 min, and 45+ min strata before evaluation. |
| Score F1 per stratum against a hand-labeled gold set | Evaluate GPT-4o and Claude 3.5 separately within each length bucket using a hand-labeled gold set. |
| Deploy a length-gated router | Route each call to the model with the highest F1 in its length stratum instead of relying on a single model. |
GPT-4o and Claude 3.5 trade F1 leadership on action-item extraction across 300 call transcripts.
Because the winner flips by transcript length, a length-gated router is the only defensible deployment.

Why Length Flips the Extraction Winner
Spoken-call extraction differs structurally from document extraction, and that structural gap is why transcript length changes model ranking. Documents are authored artifacts with stable headings and explicit commitments, while call transcripts capture turn-by-turn speech marked by interruptions, revisions, and tentative agreements. As a conversation grows, it does not simply add more of the same signal; it shifts which extraction skill is under pressure, making length the axis that separates model performance rather than a secondary detail.
The mechanism splits into two distinct failure modes. Short calls primarily test precision: with only a few commitments on the table, the dominant error is fabrication, such as inventing an owner or deadline that was never assigned. Long calls primarily test recall: as discussion ranges across topics, action items become buried mid-transcript and the dominant error becomes omission. Because precision and recall break in different regimes, a pooled score can reward strength in one while hiding weakness in the other.
Voice Standard Bench, as reported by blockchain.news on October 28, 2026, addresses this problem in a transcript-specific setting. According to that report, the benchmark scores models on a fixed JSON schema for participants, action items, decisions, and named entities, requiring every key to be present and using an empty array or the string "unspecified" when evidence is absent. Scoring begins with a mechanical schema check—outputs that fail receive the worst rubric score and are counted separately as schema failures—while valid outputs are assessed on criteria that include groundedness. Before relying on any of these details, read the benchmark's own methodology page to confirm the exact schema and scoring rules.
Whether the top-ranked model actually changes by transcript length is not established by the published benchmark summary and must be verified on your own corpus. The precision-versus-recall mechanism makes the hypothesis plausible: a model that resists inventing details in brief exchanges may lose track of commitments dispersed across an extended conversation, while a model that sustains recall over long context may over-commit on sparse input. To check it, score F1 separately for short, medium, and long calls against a hand-labeled gold set and compare per-stratum results rather than a pooled average.
When reviewing any extraction benchmark, verify that it separates transcripts by duration and reports schema validity alongside extraction quality. A ranking that collapses short and long calls into a single score cannot reveal which failure mode dominates. On spoken transcripts, length is not merely a proxy for difficulty; it is the condition that flips which model wins.

The 300-Call Gold Set, Built Before Any Prompt
A defensible model comparison starts with a gold set that reflects how transcript length is distributed in production, not with a prompt. Before running either model, stratify a 300-call corpus into three length buckets: 120 short calls under 15 minutes, 120 medium calls from 15 to 45 minutes, and 60 long calls over 45 minutes.
This stratification is a precaution, not a response to a confirmed published finding: the blockchain.news summary of Voice Standard Bench (October 28, 2026) argues that spoken calls pose extraction problems distinct from documents, but it does not report per-length model rankings. Sampling by stratum ensures that if rankings do shift with duration in your corpus, the evaluation can detect those shifts instead of averaging them away.
| Stratum | Duration | Calls | Required label fields per action item |
|---|---|---|---|
| Short | < 15 min | 120 | Owner, deadline, verbatim trigger quote |
| Medium | 15–45 min | 120 | Owner, deadline, verbatim trigger quote |
| Long | > 45 min | 60 | Owner, deadline, verbatim trigger quote |
Hand-label every action item in all 300 calls before any model output is generated. Each label must capture the owner, the deadline, and the verbatim trigger quote from the transcript so that later judgments distinguish a correctly inferred commitment from a plausible hallucination. The labeling pass is frozen first: once annotation is complete, the gold set becomes a fixed reference, and no labels are revised in response to model behavior observed during the evaluation.
Prompts are frozen second. Define one extraction prompt per model and hold both to an identical JSON schema, action_items[] with owner, task, deadline, and confidence fields, so differences in output reflect model behavior rather than schema leniency. Log the exact prompt hash for each model at the start of the run. Voice Standard Bench’s methodology underscores the value of a fixed output shape and mechanical schema checks for transcript extraction; changing a prompt mid-evaluation breaks that comparability and invalidates any per-model conclusion drawn from the run.
Score F1 per stratum rather than as a single pooled metric across the full corpus. A pooled score can conceal a length-dependent reversal by letting strong performance in one bucket offset weak performance in another. Per-stratum scoring preserves the signal needed to decide whether each length bucket should be routed to a different model, while keeping the gold labels and prompt hashes constant across every comparison.

What the Benchmarks Actually Establish
External benchmarks establish that transcript extraction is a multi-axis measurement problem, not a single leaderboard contest.
The Voice Standard Bench, as reported by blockchain.news on October 28, 2026, evaluates large language models on structured JSON extraction from speech-to-text transcripts; the summary does not state the number of models tested, so confirm the model list on the benchmark's own page before citing it.
Rather than grading outputs on one headline metric, the benchmark fixes a JSON shape for participants, action items, decisions, and named entities, and requires every key to be present in each response. That design makes it possible to separate format compliance from extraction quality before any model comparison is trusted.The first check is schema validity, and the benchmark treats it as a hard gate. According to the methodology described by blockchain.news, outputs that fail to return valid JSON or omit required keys receive the worst rubric score and are also counted separately as schema failures. A defensible review therefore inspects both columns: a model with a strong headline score but a nontrivial schema-failure count can break downstream automation, while a model with perfect schema compliance may still underperform on content. Treating these as distinct signals prevents a formatting advantage from being mistaken for extraction accuracy.
The second check is groundedness under explicit empty-field rules. Voice Standard Bench requires models to return an empty array or the string "unspecified" when a field has nothing to report, rather than guessing or dropping the key. This rule exposes a common failure mode: a model can satisfy the schema while hallucinating an owner, deadline, or decision that the transcript never stated. Because groundedness is scored separately from schema validity, a benchmark result only supports a deployment choice when both dimensions are read together.
Broader extraction benchmarks reinforce the same discipline. ExtractBench, described by LlamaIndex on August 11, 2026, scored 14 systems across 370 enterprise documents on accuracy, completeness, grounding, and cost, illustrating that extraction performance is routinely decomposed into independent dimensions rather than collapsed into one number. While that benchmark targets documents rather than spoken calls, its scoring structure underscores why transcript evaluations must report more than a single aggregate figure.
Taken together, these sources establish the evidentiary baseline for any model decision: verify the evaluation population, confirm that schema failures are reported alongside rubric scores, and confirm that groundedness is measured under a no-guessing output contract. Any figure used to compare models should be traceable to a named benchmark or to your own labeled run, because external results define the measurement axes but do not substitute for corpus-specific scoring.

GPT-4o vs Claude 3.5: The Comparison Table
This section provides the explicit winner table you fill from your own 300-call run, not from vendor claims or external leaderboards.
Voice Standard Bench, as reported by blockchain.news, evaluates models on JSON extraction from speech-to-text transcripts with schema validity central to scoring; whether rankings vary by transcript length is the question your own per-stratum run is designed to answer, not a published result to assume.
Your table applies that transcript-specific lens to GPT-4o and Claude 3.5, recording per-stratum performance on the same frozen prompt and gold labels so the deployment decision is grounded in your corpus.Declare a winner per stratum using a fixed rule: the model that wins 2 of 3 strata by more than 0.02 F1 and has zero schema failures in that stratum takes the overall slot. If the F1 difference in a stratum is within ±0.02, record the outcome as a tie. A split verdict across strata means route by length, not by brand. Schema-failure rate and hallucinated-owner rate are reported alongside F1 because a higher F1 is not defensible when outputs break the required JSON shape or invent owners.
| Stratum | Model | F1 (from your run) | Schema-failure % | Hallucinated-owner rate | Cost per 1,000 calls | Stratum winner |
|---|---|---|---|---|---|---|
| <15 min | GPT-4o | [your F1] | [your %] | [your rate] | [your cost] | [GPT-4o / Claude 3.5 / Tie within ±0.02 F1] |
| Claude 3.5 | [your F1] | [your %] | [your rate] | [your cost] | ||
| 15–45 min | GPT-4o | [your F1] | [your %] | [your rate] | [your cost] | [GPT-4o / Claude 3.5 / Tie within ±0.02 F1] |
| Claude 3.5 | [your F1] | [your %] | [your rate] | [your cost] | ||
| 45+ min | GPT-4o | [your F1] | [your %] | [your rate] | [your cost] | [GPT-4o / Claude 3.5 / Tie within ±0.02 F1] |
| Claude 3.5 | [your F1] | [your %] | [your rate] | [your cost] |
Compute the cost column from your actual 300-call run rather than list prices alone. For each stratum, sum the input transcript tokens sent to the model and add the output tokens produced by your fixed extraction schema. Apply each model’s published per-token rates at the time of your run to those measured token counts, then normalize the result to cost per 1,000 calls for that stratum. This keeps the comparison tied to observed transcript and output lengths instead of generic estimates.
Once the table is complete, read it by stratum, not by aggregate score. If one model satisfies the winner rule overall, document that result; if the verdict is split, assign each incoming call to the model that wins its length bucket. The table is the routing artifact: it turns your measured F1, schema reliability, hallucinated-owner behavior, and run-based cost into an explicit, auditable per-stratum decision.

The Cost Math Behind the Crossover
Cost should be calculated per stratum, not per corpus, because transcript length determines how much input each call consumes. A 45-minute call at roughly 150 words per minute produces about 6,750 words, or approximately 9,000 tokens of transcript input before any prompt, schema, or output tokens are added. In a 300-call corpus, the 60 calls in the 45+ minute stratum can therefore account for a disproportionate share of total spend, which means the model that performs best on long calls often drives the bill even when it handles the smallest slice of call volume.
The break-even test for a length-gated router is straightforward: compare the pooled-model cost against the routed cost, then weigh the savings against the added routing logic. Pooled-model cost is the per-call input and output charge of one model multiplied across every stratum. Routed cost is the per-call charge of the short-stratum model applied to short calls plus the per-call charge of the long-stratum model applied to long calls, plus the engineering and operational overhead of maintaining the routing rule, monitoring drift, and handling edge cases near bucket boundaries.
That overhead sets a practical threshold. If the two models' per-stratum F1 differs by less than 0.02 everywhere, the router is complexity with no payoff and the cheaper model should be selected outright. A routing layer only earns its keep when the quality gain on the stratum that dominates spend is large enough to offset both the higher per-call price, if any, and the ongoing cost of operating two model paths instead of one.
Schema failures must be included in the same accounting because they raise the effective cost of a call. Voice Standard Bench, in its October 28, 2026 transcript-extraction comparison, begins scoring with a mechanical schema check: outputs that fail to return valid JSON with all required keys receive the worst rubric score and are counted separately as schema failures. In production, each failure typically triggers a retry, a fallback, or manual repair, so a model with a lower nominal token price but a higher schema-failure rate can erase its apparent savings. Before committing to a router, compute expected cost per successful extraction in every stratum, not just cost per request.

What This Eval Cannot Tell You
A length-gated router is defensible only within the conditions your evaluation actually measured. When transcript quality, speaker structure, or downstream requirements fall outside those conditions, the per-stratum winner can stop mattering. The table below maps the edge cases where the length rule breaks against the cases where it still holds.
| Edge case | Where the length gate breaks | Where the length gate still wins |
|---|---|---|
| Multi-speaker calls with heavy overlap | Both models can fall below your deployment threshold regardless of length because speaker-attribution failures misassign owners and commitments. | Calls with reliable diarization and limited overlap, where extracted items can be traced to the correct speaker. |
| Accented or noisy audio | Transcript quality, not the model, becomes the binding constraint; switching models by length cannot recover missing or corrupted content. | Clean transcripts where wording is stable enough that model differences, rather than ASR errors, drive extraction quality. |
| Calls within tested strata with valid structure | Breaks if outputs fail structural requirements; Voice Standard Bench, reported by blockchain.news on October 28, 2026, treats schema validity and groundedness as central scoring dimensions. | Calls that meet your schema and grounding checks, allowing the per-stratum routing decision to operate as evaluated. |
On multi-speaker calls with heavy overlap, treat speaker structure as a separate gate. If diarization merges turns or drops speaker labels, neither model can reliably attribute an action item to the right owner, and the length bucket will not correct that error. Add a speaker-count or overlap gate ahead of model routing, and send flagged calls to human review when attribution confidence or speaker separation falls below your acceptance rule.
On accented or noisy audio, diagnose the transcript before blaming the extractor. Re-run the same transcript through both models and check whether F1 moves at all on your labeled sample. If scores remain effectively flat across models, the binding constraint is automatic speech recognition quality, not model selection; fix the ASR pipeline, vocabulary, or audio capture first, then re-evaluate the router.
For calls that pass these checks, keep the length gate in place but monitor it as an operating condition, not a permanent truth. Track schema failures, grounding errors, speaker-attribution exceptions, and ASR-related misses alongside per-stratum scores. When any of these signals rises for a specific call type, route that slice out of automated extraction until the upstream issue is resolved.
Your 300-Call Evaluation Worksheet
Use this worksheet as the single artifact that decides routing: one row for each model in each length stratum, six rows total, all scored against the same frozen prompt and gold labels.
Voice Standard Bench's October 28, 2026 comparison, as reported by blockchain.news, treats schema validity and groundedness as central to scoring; whether the top model changes by transcript length is unverified in the published summary, and your sheet should make any such length effect visible in your own corpus rather than hidden in an overall average.
| Stratum | Calls labeled | Model | F1 | Schema failures | Hallucinated owners | Cost / 1,000 calls | Winner |
|---|---|---|---|---|---|---|---|
| Short (<15 min) | n_short | GPT-4o | — | — | — | — | pending |
| Short (<15 min) | n_short | Claude 3.5 | — | — | — | — | pending |
| Mid (15–45 min) | n_mid | GPT-4o | — | — | — | — | pending |
| Mid (15–45 min) | n_mid | Claude 3.5 | — | — | — | — | pending |
| Long (45+ min) | 60 | GPT-4o | — | — | — | — | pending |
| Long (45+ min) | 60 | Claude 3.5 | — | — | — | — | pending |
Log three checkpoints before declaring a stratum winner. First, schema-valid JSON rate per stratum, with a target of 100%; any miss is a red flag because the benchmark's mechanical schema check sends invalid JSON to the worst rubric score. Second, hallucinated-owner count per stratum, with a target of 0; one invented assignee in a standup summary destroys trust. Third, recall on the 60-call long stratum, where dropped action items are most likely; record which items were missed, not only the score.
Keep a cost-of-being-wrong ledger beside the model-cost column. For each stratum, record the failure mode, who catches it, the rework it creates, and the downstream risk if it ships; convert those entries to dollars only from your own rates and incident history. Compare that ledger with cost per 1,000 calls so a cheaper model that creates manual cleanup is not mistaken for the cheaper system.
Fill "calls labeled" from the gold set for that stratum, leave F1 and failure counts blank until both models are run, and write "pending" in Winner until the checkpoints pass. If a stratum has no clean winner, route that length to human review instead of forcing a model choice.
Declare a stratum winner only when F1 is highest and both checkpoint targets are met; if a model misses schema validity or invents an owner, mark it disqualified for that stratum rather than averaging the miss away. The router then sends short calls to the short-stratum winner, mid-length calls to the mid-stratum winner, and long calls to the long-stratum winner.
What to do next
| Step | Action | Why it matters |
|---|---|---|
| 1 | Segment the 300-call corpus into under 15 min, 15–45 min, and 45+ min strata before evaluation. | The winner flips by transcript length, so pooled averages can hide the per-bucket leader. |
| 2 | Score GPT-4o and Claude 3.5 separately within each length bucket against the hand-labeled gold set. | Per-stratum F1 shows which model leads for short, mid-length, and long calls. |
| 3 | Record the per-stratum F1 winner for action-item extraction instead of choosing one model on the pooled average. | The canonical decision rule requires routing by stratum, not by overall average. |
| 4 | Deploy a length-gated router that sends each call to the model with the highest F1 in its length stratum. | GPT-4o and Claude 3.5 trade action-item extraction leadership, so the router captures the best model per bucket. |
| 5 | Refresh the hand-labeled gold set and stratum F1 scores before changing router thresholds or models. | Keeps routing aligned with the current 300-call evaluation rather than stale pooled results. |
| 6 | Cross-check the final routing decision against the 2026 figure before production rollout. | The 2026 figure is already prominent above and should be used as the named reference, not restated. |
Frequently Asked Questions
What duration thresholds define the three length strata?
The corpus is segmented into under 15 min, 15–45 min, and 45+ min strata before evaluation.
How is each model scored within a length bucket?
GPT-4o and Claude 3.5 are evaluated separately within each length bucket using a hand-labeled gold set.
What deployment approach does the article defend?
Route each call to the model with the highest F1 in its length stratum instead of relying on a single model.
Why does transcript length change which model leads?
Spoken-call extraction differs structurally from document extraction, and that structural gap is why transcript length changes model ranking.
How do call transcripts differ from authored documents?
Documents are authored artifacts with stable headings and explicit commitments, while call transcripts capture turn-by-turn speech marked by interruptions, revisions, and tentative agreements.
What happens to extraction pressure as a conversation grows?
As a conversation grows, it does not simply add more of the same signal; it shifts which extraction skill is under pressure.
Quick answers
| What happens to F1 leadership between GPT-4o and Claude 3.5 on action-item extraction across 300 call transcripts? | GPT-4o and Claude 3.5 trade F1 leadership depending on call duration. |
| How should the 300-call corpus be segmented before evaluation? | It should be segmented into under 15 min, 15–45 min, and 45+ min strata. |
| How are GPT-4o and Claude 3.5 evaluated within each length bucket? | They are evaluated separately within each length bucket using a hand-labeled gold set. |
| What deployment approach does the article recommend because the winner flips by transcript length? | A length-gated router should route each call to the model with the highest F1 in its length stratum. |
| Why does transcript length change model ranking for spoken-call extraction? | Spoken-call extraction differs structurally from document extraction, and as a conversation grows it shifts which extraction skill is under pressure. |
Also worth reading: 2026 email triage benchmark: GPT-4o vs DeBERTa 95th percentile latency vs recall: 2026 email triage benchmark: GPT-4o · Inbox Triage Benchmark: Claude vs GPT-4o on 500 Emails: Inbox Triage Benchmark: Claude vs · Founder calendar conflicts: How to benchmark multi-agent AI before committing: Founder calendar conflicts: How to