Inference Pricing for Small Teams: What 3,065 Weekly Agent Runs…

Every Sunday I pull the job telemetry off my agent stack. Last week the number was 3,065 runs. The week before, roughly the same. The week before that, roughly the same. That flatness is the interesting part, because it means I now have a stable enough sample to answer a question I'd been getting wrong for months.

The question: does inference pricing for small teams actually track the headline per-token rates that providers announce? My answer after 3,065 runs a week: almost not at all. What tracks is retry and failure volume. A price cut on a model you rarely call changes nothing. A flaky tool call that forces three retries on a model you call constantly changes everything.

I run this stack solo. Hermes Agent owns Telegram, Discord and cron; the Claw brain and the workspace scripts sit behind it. Most of those 3,065 runs are boring: classify this, extract that, summarize a thread, check a price, write a short status line. A minority are hard: multi-step planning, code review, anything where a wrong answer costs me an hour of debugging.

That split is the whole story.

The 3,065 number is a failure-rate number, not a volume number

Here's what I got wrong early on. I treated run count as a proxy for spend. It isn't. Run count is a proxy for opportunity to fail, and failure is where the money goes.

An agent run that succeeds on the first pass costs one call. A run where the model emits malformed JSON costs a call, a parse error, a re-prompt with the error appended, and usually a second attempt. If the tool itself times out after that, you've paid for tokens that produced nothing usable. The retry is not a discount. The retry is full price for work you already bought once.

OpenRouter's synthesis of recent 2026 writeups puts it plainly: for agentic systems, retry and failure volume drive inference spend far more than headline per-token rates [11]. The same synthesis notes that recent writeups quantify the gap with concrete multipliers rather than anecdotes [12]. That matches what I see in my own logs. The runs that hurt are not the expensive-model runs. They're the cheap-model runs that wouldn't converge.

So I stopped asking "which model is cheapest per token" and started asking "which model has the lowest cost per completed run." Those are different questions with different answers, and the second one is the only one that shows up on a bill.

Takeaway: measure cost per completed run, not cost per token. The gap between them is your retry tax, and it's usually bigger than any pricing difference between tiers.

Where the hidden cost actually concentrates

Three places, in descending order of how much they cost me.

Structured output that the model almost gets right. Extraction jobs are the worst offenders. The model returns something that looks like JSON, parses at the top level, and fails validation two fields deep. You burn a full generation to discover the shape is wrong. This is where cheap-tier models earn their keep and where they can quietly cost more than a premium model, because a premium model that returns valid structure on the first pass is cheaper than a cheap model that needs three passes.

Tool calls that fail for environmental reasons. A timeout, a rate limit, a network blip. The model did its job; the world didn't cooperate. You pay for the generation, get nothing, and pay again. This is the failure mode that has nothing to do with model quality, which is exactly why it's easy to miss when you're shopping on benchmarks.

Long-context retries. When a retry has to re-send the full conversation plus the error message, the second call is more expensive than the first. Retries don't just double your call count. They inflate the input side of every subsequent call.

The Kubernetes contributor docs make a related point about how testing and contributor-experience work intersect [4]. The parallel I'd draw: your retry behavior is a contributor-experience problem for your own agent. If the path to a successful run is fiddly, you pay for the fiddliness on every single run, forever.

Takeaway: audit your failures by category before you audit your model choice. Environmental failures and near-miss structured output are usually the two biggest line items, and neither is fixed by switching models.

Why cheap-by-default is still right, with one correction

I route routine work to the cheapest capable tier and reserve strong reasoning for the small set of tasks where a better answer changes the outcome. That principle holds. The 3,065 runs confirmed it, with a correction I didn't expect.

The correction: cheap-by-default only works if you also make failure cheap. Routing extraction to a small model is correct when the small model reliably produces valid structure. It's incorrect when it produces almost-valid structure, because almost-valid is the most expensive possible outcome. You pay for the generation, you pay for the validation, and you pay for the retry.

So the tiering rule needs a second clause. Route by consequence of failure, not just by difficulty of task.

The extraction row is the one that surprised me. It looks like a cheap-tier job. It behaves like an expensive one when the schema is strict. The fix isn't a better model, it's a stricter contract: validate before you retry, and fail fast when the shape is wrong so you don't pay for a third attempt on a prompt that was never going to work.

Takeaway: tier by consequence of failure. A low-difficulty job with a high failure cost belongs in the same bucket as a hard job, and the fix is usually a tighter output contract rather than a bigger model.

What the open-weights gap means for a stack this size

The open-weights versus closed gap keeps narrowing where it matters for cost. For a builder at my scale, the useful question isn't "which model is best." It's "at what capability threshold does the cheap open-weights option become good enough that its failure rate stops eating the savings."

Recent llama.cpp builds show how fast the open side moves. Build b11095 lands a Hexagon HMX-optimized GATED_DELTA_NET, with the commit notes admitting HMX support is working but "not-pipelined and slow for now" [7]. Build b11094 is a dependency bump to cpp-httplib 0.57.1 [8]. Neither is a headline. Both matter, because the open-weights option gets cheaper and faster on a weekly cadence, and that cadence is what makes the threshold move.

On the closed side, the SDK churn is constant. openai-python v3.17.0 added external storage configuration management [6]. LangChain shipped langchain-openai 1.6.3, which exposes inferred Responses API routing at initialization [9], and langchain-core 1.6.4, which deprecates chat message history [10]. These are plumbing changes, and plumbing changes are where retry behavior gets better or worse without anyone announcing it.

Here's the practical version. I keep the expensive model as a small, swappable minority of calls. When the open-weights option clears the bar for a job class, I swap it in and watch the failure rate, not the token price. If the failure rate holds, I keep it. If it climbs, the savings were fictional and I revert.

# The only pricing metric I trust: cost per completed run
# Pull last week's runs, split by outcome, divide spend by successes

awk -F',' '$3=="success" {s+=$5; n++} $3=="failure" {f+=$5; m++}
 END {
 printf "completed runs: %d\n", n
 printf "failed runs: %d\n", m
 printf "spend on successes: %.4f\n", s
 printf "spend on failures: %.4f\n", f
 printf "cost per completed run: %.6f\n", (s+f)/n
 }' runs.csv

That last line is the number I make decisions on. Not the per-token rate. The per-completed-run rate, which bakes in the retry tax automatically.

Takeaway: judge a model swap on failure rate, not token price. A cheaper model with a higher failure rate is a more expensive model wearing a discount.

What's Next

Three things I'm testing over the next few weeks.

First, a hard validation gate before any retry. If the output fails schema validation, the run dies instead of re-prompting. I want to see whether killing runs early costs less than nursing them back to health. My guess is yes, because the third attempt on a broken prompt is pure waste.

Second, per-job-class failure budgets. Not a global ceiling, but a ceiling per job type, checked before the call rather than after. A cost ceiling that gets checked after the charge is a post-mortem, not a control.

Third, a swap test on extraction specifically. I want to know whether the open-weights option at the cheapest tier holds its failure rate on strict-schema extraction, or whether that's the one job class where the premium model is genuinely cheaper per completed run. I don't know the answer yet. That's the point of running it.

If you're running an agent stack of your own, I'd like to know where your retry tax concentrates. My guess is it's not where you think it is. Mine wasn't. Pull your logs, split successes from failures, and divide spend by successes. Tell me what number you get, and whether it changes which model you'd pick.

Related Reading

• Vector databases moved from RAM to lakes — here's why it matters — I hit a wall trying to scale our internal retrieval system last month. We were using a managed vector DB that charged per GB of RAM, and our embedding corpus…
• Calendly vs AI Booking Agent for Coaches: What Books More — Calendly books whoever picks a slot. An AI booking agent asks a few qualifying questions first, so the discovery calls that land on your calendar are worth...
• What Is an AI Booking Agent for Clinics & Studios? (And What It Isn’t) — Your front desk is drowning in after-hours inquiries and missed callbacks, and a parent searching for a pediatric dentist at 9 p.m. doesn't wait to find out…