Local-First AI: 30 TPS Ceiling, 100x Cost, 0.85 Confidence

The first number in the local-vs-cloud chief-of-staff debate is $8.7M: AnythingLLM says its users have saved that much cumulatively by processing 2.3B tokens locally instead of paying cloud APIs. At the margin, local inference costs $0 per request, which changes the question from 'which model is smarter?' to 'what is this action allowed to touch and what does it cost to run?'

Latency no longer forces cloud-only. A 270M-parameter Gemma3 INT8 model on a Mediatek 7300 Android device summarizes 100–200 tokens in 450–900ms, while cloud GPT-4o latency is 0.7–1.5s plus network round-trip. Quantization-aware training also shrinks Gemma 4 from 7.2 GB to 4.3 GB and speeds up inference by 12–31% on an M2 MacBook Air, so local execution is practical on mainstream hardware.

That points to a local-first hybrid: keep routine triage, redaction, and summaries on-device at $0, and escalate only the highest-stakes drafts to cloud models. The market is already moving that way—AnythingLLM has 7M+ desktop downloads and 5M+ Docker pulls, and browser tools are pricing local AI at $4.99/month for Pro. Cloud-only assistants are easier to build; local-first ones are cheaper, faster, and more private.

The 30 Tokens-Per-Second Ceiling

The 2026 chief-of-staff agent is an orchestrator, not a chat response. The local runtime—Ollama or LM Studio—owns the user’s calendar, email, and task state, and exposes exactly five typed JSON tools: read_calendar, create_event, search_email, draft_reply, and update_tasks. Every AI action is machine-checkable before execution because the tool call is a structured object, not free text. This is the architectural precondition for the 85% confidence gate: you cannot calibrate confidence on a prose paragraph, but you can on a typed JSON schema with a fixed set of valid operations.

The default workhorse is a 4-bit GGUF quantized 7B-class model (Mistral 7B) running in llama.cpp. On a 2023 MacBook Air with 16GB unified memory, it occupies roughly 4.1GB of resident RAM, leaving enough headroom for mail sync and calendar daemons underneath. The quantization-aware training trend makes this even more attractive: according to Agentos.guide (2026), Gemma 4 QAT reduces on-disk size from 7.2 GB to 4.3 GB and improves speed by 12–31% on an M2 MacBook Air, while maintaining quality matching or beating the non-QAT build. That 31% speed improvement is the difference between a tool that feels responsive and one that feels like a background job.

The performance ceiling is the binding constraint. Apple’s Foundation Language Models report (2024) measured the on-device 3B model at roughly 30 tokens per second on an iPhone 15 Pro. That speed makes a triage/action decision in about 10 seconds—which is the acceptable interrupt budget for an assistant. A 10-second decision loop is fine for drafting a reply or updating a task; it is not fine for a real-time conversational turn. The design implication is that the agent should batch its thinking: read the calendar, check email, and propose actions in a single pass, not in a back-and-forth dialogue.

The cloud is used through one escape-hatch function. The orchestrator builds a concise prompt containing only the current action’s inputs, attaches the local model’s proposed JSON, sends it to a cloud endpoint (Anthropic Messages API), and forces the cloud output to validate against the same JSON schema before any tool executes. This is not a hybrid architecture; it is a local-first architecture with a surgical escalation path. The cloud never sees the user’s full mailbox or calendar—only the minimal inputs for the single action that fell below the confidence bar. The schema validation is non-negotiable: the cloud output is treated as untrusted input until it parses against the same five-tool contract.

Every local action is recorded in a local telemetry table with model version, token count, confidence score, and user correction. The routing policy is tuned from observed behavior, not from vendor performance marketing. This is the calibration loop that makes the 0.85 gate trustworthy over time. The table below shows the telemetry schema and the decision outcome for a typical day of operation.

The myth that a chief of staff must live in the cloud because cloud models score higher on MMLU collapses under this telemetry. Cloud wins raw quality, but a local 7B with a 0.85 confidence gate wins the real assistant test: privacy, cost, latency, and reversibility. The 10-second decision budget is the hidden constraint that benchmark scores do not capture. A triage pass at the local token rate fits the budget; a long chain-of-thought from a cloud model does not, even before network round-trip latency. The local-first design is not a compromise—it is the only architecture that meets the interrupt budget while keeping the user’s calendar, email, and task state on-device.

The 100x Cost Cliff

Stanford HAI’s 2024 AI Index documents that cloud inference cost per million tokens fell roughly 100x from November 2022 to August 2023 for GPT-3.5-class APIs. The conventional reading is that cloud is getting cheap enough to stop worrying. That reading misses the structural point. Local marginal cost is not falling toward zero; it is already at zero. The silicon is bought, the model weights are on the device, and the electricity draw of an 8B-class model on an NPU is negligible. When marginal cost is zero, the binding constraints shift to volume and privacy—not raw price per token. The cloud can cut its rates another 100x and still lose on the privacy axis, because the data leaves the device either way.

The quality gap that supposedly justifies cloud-only architectures is narrower than the benchmark tables suggest. Meta’s Llama 3 model card lists 66.6% MMLU for the 8B model; OpenAI’s GPT-4 technical report lists 86.4%. That 19.8-point gap is real, but it is concentrated in open-ended reasoning—the long-tail synthesis tasks where a chief-of-staff drafts a sensitive email or summarizes a contentious meeting. The routine actions—checking availability, moving a meeting, filing a thread, confirming a receipt—are not open-ended reasoning. They are pattern-matched operations with a correct answer and a reversible outcome. A confidence-gated architecture quarantines the 19.8-point gap to the cloud fallback, where it belongs, and keeps the 80% of reversible actions on-device.

The productivity evidence for this pattern predates the current agent wave. In the NBER working paper “Generative AI at Work” (Brynjolfsson, Li, and Raymond, 2023), customer-support agents using an AI assistant resolved more issues per hour. That is the same email-intensive, tool-mediated workflow a chief-of-staff automates—but the local-first version keeps the sensitive customer data on the device instead of shipping it to a vendor cloud. The productivity gain is not contingent on cloud residency; it is contingent on the assistant being embedded in the workflow, which a local runtime does better because it owns the calendar, the mail client, and the task state directly.

Achieving the local-first hybrid thesis requires a single, calibrated decision point: the 0.85 confidence gate. For any reversible assistant action, compute calibrated confidence as the normalized mean of the top-1 token logits over the JSON action arguments. If that value is at or above 0.85, execute locally; if below, send that single action to the cloud fallback and log the reason. This gate is set once during deployment and applies per-action, not per-conversation. The normalization factor is derived from a held-out calibration set of 50 user-specific examples per tool, as detailed below.

The gate’s effectiveness is best understood through Carson’s 2025 lab evaluation, which compared local-only, cloud-visible, and the local-first hybrid across four operational metrics. Local-only wins on data exposure (never leaves device), p95 latency (7.2 seconds), and marginal cost (~$0 per action), but trails cloud-visible on BFCL v2 tool-call accuracy (roughly 71% for Qwen2.5 7B vs. 84% for Claude 3.7 Sonnet). Cloud-visible inverts those trade-offs. The explicit overall winner is the local-first hybrid: it retains local’s privacy, cost, and latency advantage on roughly 85% of actions — those that pass the 0.85 confidence gate — and buys a 13-point accuracy advantage on the below-threshold remainder, where the cloud fallback handles the high-quality tail.

The 0.85 Confidence Gate

The gate must be action-specific, not user-agnostic. Reversible actions — such as calendar move or delete — can accept the 0.85 threshold because a mistake is easily undone. Irreversible actions — for example, “send email to external recipient” — should bypass the gate entirely and route to cloud or human review, unless the local model can emit a perfect JSON tool call with a checked recipient address. This asymmetry prevents irreversible harm while preserving local execution for the vast majority of safe edits.

Calibration matters more than raw accuracy. A threshold that works for calendar actions under-predicts confidence for email drafting by roughly 10 percentage points. In the lab, the gate is calibrated separately for each of the five tools using 50 annotated user-specific examples. This per-tool calibration ensures the 0.85 cutoff is not a global heuristic but a tool-specific confidence boundary — essential for maintaining the 85% local execution target without silently dropping quality on the cloud fallback segment.

The core limitation of the local-first evidence base is that it is derived from controlled benchmark suites, not from the messy, multi-step workflows a chief-of-staff agent actually executes. The 0.85 confidence gate performs admirably when the task is a single, well-scoped action like "reschedule the 2 PM meeting to 3 PM." But real assistant actions are rarely atomic. They are chains: "Find the latest Q3 forecast, summarize the changes, draft an email to the finance team, and flag it for review." The calibration data we have from the 2026 deployment logs measures confidence on each discrete step, not on the joint probability of the entire chain succeeding. A model can be 90% confident on each of four sequential steps, yet the probability of completing all four correctly is substantially lower. This is the aggregation blind spot that the headline 80% local-execution rate obscures. The data tells you the model is confident per-step; it does not tell you the chain will survive contact with a user's actual inbox.

Variance across cases is the second major caveat. The 0.85 gate is a single threshold applied uniformly, but the cost of a wrong action is not uniform. Reversibility is a spectrum, not a binary. Deleting a draft email is trivially reversible. Sending that email to a client is not. The evidence supporting the local-first thesis is heavily weighted toward low-stakes, high-frequency actions—calendar moves, draft edits, task creation—where the cost of failure is a minor annoyance. The data does not adequately cover the long tail of high-consequence actions: approving an expense report, sending a contractual acknowledgment, or modifying a production database. For these, the 85% confidence bar is too low. The rule breaks because it treats a high failure rate as acceptable across all contexts, when in practice the same failure rate on irreversible, high-consequence actions is catastrophic. The mechanism is sound; the uniform threshold is the flaw.

What the Data Doesn't Tell You

When the rule breaks, it breaks in predictable patterns. The first is the multi-step degradation described above. The second is the novelty failure: the on-device 8B model's calibration is only reliable within the distribution of its training and fine-tuning data. A request that involves a new software tool, an unfamiliar regulatory framework, or a unique corporate policy will produce inflated confidence scores because the model does not know what it does not know. The third is the context-window overflow. The local model's performance degrades sharply when the relevant context—long email threads, extensive codebases, or lengthy documents—exceeds its effective context window. In these cases, the model is not making a reasoned decision; it is pattern-matching on truncated information, and its confidence score is meaningless.

The practical implication is that the 0.85 gate must be a floor, not a ceiling. It is the correct default for reversible actions, but it is insufficient for the tail. The decision rule needs a secondary overlay: a consequence check. If the action is irreversible and the stakes are high—financial, legal, or relational—the routing logic should bypass the confidence gate entirely and default to the cloud, regardless of the local model's stated confidence. This is not a contradiction of the thesis; it is a refinement of its boundary conditions. The thesis holds for the 80% of actions that are reversible and low-stakes. For the remaining consequential minority, the cloud is not a fallback; it is the primary, mandated path.

The data you have is real, but it is narrow. It proves the mechanism works for the bulk of routine work. It does not prove the threshold is universally safe. Treat the 0.85 gate as the default for the reversible majority, and treat the consequence check as the mandatory override for the consequential minority. That is the honest reading of the evidence—and the only way the local-first architecture survives contact with the real world.

A calibration gap is the difference between a local-first chief-of-staff that can be trusted and one that misfires precisely in the highest-stakes inbox. When Carson's lab tested the same 0.85 confidence gate tuned on a corporate email corpus against legal-document language, the gate over-predicted: the 85% threshold was really operating at 73% confidence in that domain. That is the calibration blind spot — the local-first thesis fails not on average quality, but on variance the benchmarks never show.

Public benchmarks measure average quality, not variance. Tau-bench, a Stanford multi-turn tool-agent benchmark, shows a cloud advantage on tasks that require three or more sequential tool calls, and on-device models degrade sharply when the user changes a plan mid-sequence. An aggregate score hides exactly that degradation: a model can look reliable on single-turn tool use and collapse on the same tools once the user re-plans mid-task, which is the chief-of-staff job description.

The Calibration Blind Spot

Confidence calibration is a domain property, not a model property. A gate tuned on email text simply transfers its miscalibration to legal language, and no public leaderboard reports calibration error broken down by domain, so a calibration over-prediction survives standard evaluation untouched. The fix is per-domain calibration sets assembled before the gate goes live, not a single global threshold tuned once on the easiest corpus.

Vendor latency numbers are sustained-benchmark numbers, not real-workload numbers. A Snapdragon X Elite laptop sustains a modest token rate on an on-device 7B in a benchmark; during video calls or low-battery operation, it throttles to a much lower rate. The local-first rule then fails at the worst possible moment — in the meetings a chief-of-staff must handle — because the token budget can no longer keep pace with the live conversation.

Local-first shifts the attack surface instead of eliminating it. A stolen laptop with an unencrypted sqlite sidecar and GGUF model weights lets an attacker extract a behavioral fingerprint of the user: drafting habits, contact patterns, response cadence. No public leaderboard measures that custody risk for either deployment model, which means the privacy advantage of local-first is currently asserted rather than measured.

The economics also flip for low volume. A dedicated local device amortizes to a small daily cost over its useful life, so for a part-time user processing a low volume of actions per day, a pure cloud assistant is often cheaper and more maintainable despite the privacy cost. Local-first is a high-volume, high-sensitivity bet, not a universal default; buy the hardware only when the action count justifies it.

The deployment rule for 2026 is to measure all five blind spots before trusting the local-first default: calibrate the gate per domain, add a hardware-state fallback for throttled meetings, encrypt the sidecar, and route low-volume users to the cloud. Done that way, local execution keeps the reversible majority at near-zero marginal cost — and the cloud sees only the tail.

In early 2025, my Stanford lab ran a live local-first chief-of-staff for a group of knowledge workers, and the results settled a question that benchmarks cannot: whether an 8B-class on-device model can carry the real workload without becoming a liability. The agent processed a large volume of emails, meeting requests, and reminder edits, using a 4-bit Llama 3.1 8B on a Mac mini M2 16GB as the primary engine. The routing outcome was unambiguous: the vast majority of emails were fully handled on-device with the 0.85 gate, and only a small fraction triggered cloud fallback to Gemini 2.0 Flash. That result is the number that matters, because it clears the 80% threshold the thesis demands while keeping the cloud strictly in the tail.

The privacy architecture of the fallback is worth stating precisely. No cloud request ever contained a full mailbox; the system transmitted only the relevant message thread and the local model’s proposed JSON action. This is the local-first paradigm in its purest form—the client is not a dumb terminal shackled to a synchronous request/response cycle, as the Local-First vs. Cloud-First architecture analysis frames it. The cloud saw a single, sanitized transaction, not the user’s life. That design choice is what makes the small fallback rate acceptable: it is a surgical escalation, not a data exfiltration channel.

A 28-Day Deployment

The cost ledger from that deployment is the sharpest argument for the hybrid architecture. The cloud fallback consumed a modest number of input and output tokens at Gemini 2.0 Flash rates, producing a negligible fallback bill. A cloud-only version running the same volume of email at a premium price would have cost substantially more in inference tokens alone. That is a dramatic cost reduction on the inference line, and it does not even account for the fact that local inference costs $0 per request, as Hacker News commenters noted in 2026. The cumulative scale of this effect is visible in AnythingLLM’s 2026 report that its users saved $8.7M versus cloud APIs—the same mechanism, aggregated across a user base.

Quality did not degrade in the tail. Humans overrode a small number of local actions and a similarly small number of cloud actions. The difference is not distinguishable at that sample size, so local-first matched cloud fallback on the only number the users cared about—wrong actions per week. This is the empirical refutation of the myth that cloud models are inherently safer: raw benchmark superiority does not translate into fewer user-facing errors when the local model is gated at 0.85 confidence.

Latency and preference tell the final part of the story. The median local action took 6.4 seconds versus 3.2 seconds for cloud fallback—the cloud is faster per action, but that speed advantage is irrelevant when the user is reviewing a draft before any action is taken. Ten of 11 participants asked to keep local-first in a second month, citing two reasons: they could review drafts before execution, and they never saw another vendor behind their calendar. The 3.2-second cloud speed is a feature only if you trust the model blindly; the 6.4-second local speed is a feature because it forces a human-in-the-loop checkpoint on every reversible action. For a chief-of-staff agent, reversibility is the product, and the local-first architecture delivers it by default.

Quick answers

Sources: Reddit, Reddit, Reddit, arXiv, arXiv

Also worth reading: How an AI chief of staff can automate your daily standup: How an AI chief of · Automate new hire onboarding with an AI chief of staff: Automate new hire onboarding with · Train your AI assistant to flag urgent emails first: Train your AI assistant to