The Real Cost Model of LLM Inference in Production

A price per million tokens looks precise, which makes it tempting to multiply by forecast traffic and call the result a budget. Real applications pay for much more complicated paths. One user action may trigger query rewriting, embedding, retrieval, reranking, a generation call, a safety check and a retry after invalid structured output. Long system instructions and retrieved passages are paid repeatedly, while output length varies by task and model behaviour. Self-hosted models replace the provider invoice with accelerators, idle capacity, operations and queueing constraints. The useful unit is therefore cost per successful business task at an acceptable latency and quality, with the entire execution trace included.

Start with token accounting by operation

Measure input and output tokens for every model call and attribute them to a trace, feature, tenant and version. Separate fixed input, such as system instructions and tool schemas, from variable user and retrieval content. Output tokens often cost differently and determine completion latency, so averages alone are weak planning inputs. Retain percentiles and distributions by task. A small group of long documents can dominate spend. Include cached-input pricing only where requests genuinely meet the provider’s cache semantics, and record cache hits rather than assuming them. Tokenisers differ between model families, meaning character counts are not a dependable substitute when comparing providers or forecasting a migration.

Retries and loops multiply the visible call

Timeout retries, schema-repair prompts and agent tool loops create amplification. A retry may repeat the full context, not merely the failed fragment. Instrument attempt count and termination reason, and distinguish transport retries from application retries. Use idempotency keys for any tool that can change state so repeated model or network actions do not duplicate transactions. Bound loops by step count, time and spend, but also improve the first attempt: simpler schemas, validated tool descriptions and deterministic repair can be cheaper than asking the model again. The denominator matters here. If a task consumes calls but ends in failure or manual rework, those costs belong to the cost of successful tasks, not to an ignored error bucket.

• Input and output tokens for every call, including graders and safety models.
• Embedding, search, reranking, storage and network charges.
• Retry rate, tool-loop depth and invalid-output repair.
• Reserved or idle serving capacity and operational engineering.
• Human review, correction and escalation required to complete the task.

Latency has an economic shape

Time to first token affects perceived responsiveness; total completion time affects workflow throughput. Batching can improve accelerator utilisation but adds queueing delay. Larger contexts increase prefill work, while longer outputs extend decode time. Provider rate limits can turn traffic bursts into queues or failures, requiring admission control and backpressure. Model the service objective by task instead of demanding one latency target everywhere: an interactive assistant differs from an overnight classification job. Cost optimisation that causes abandonment or forces an employee to wait is not necessarily saving money. Measure concurrency, queue time, prefill, generation, tools and network separately so the expensive delay can be addressed rather than hidden behind an end-to-end average.

Hosted and self-hosted economics differ

Hosted APIs offer variable consumption, managed scaling and rapid access to model improvements, while imposing external pricing, limits and data-processing considerations. Self-hosting provides control over weights, quantisation, batching and deployment location, but accelerator cost continues when traffic is low. Capacity must cover peaks or tolerate queues; redundancy and rolling deployments add headroom. Effective tokens per second depends on model size, numeric precision, sequence lengths, batch composition and serving engine, not just the accelerator specification. Compare both options with measured workload distributions and equivalent quality. A smaller self-hosted model that needs more retries or review may be more expensive per completed task than a larger hosted model.

Optimise context before degrading the model

Repeated boilerplate and indiscriminate retrieval are common sources of avoidable tokens. Reduce tool schemas to what the task can use, retrieve fewer but better-ranked passages, summarise stable conversation state carefully and route simple tasks to specialised paths. Semantic caching can help repeated informational queries, but keys must include permissions, source versions and configuration, and unsuitable personalised responses must never cross users. Model routing should be based on evaluated task difficulty, with a fallback when the smaller model is uncertain or invalid. Output limits and concise response instructions control runaway generation, though limits must not truncate required content. Every optimisation should be rerun through quality and safety evaluation.

Forecast ranges, then reconcile them

A forecast should combine traffic scenarios with observed token and retry distributions, rather than one average request. Include price changes, growth in context size, peak capacity and a contingency for product experimentation. Tag spend at the gateway or orchestration layer and reconcile internal token records with provider bills. Unexpected movement should be explainable by volume, mix, prompt version, model route or failure rate. Budgets and alerts can operate per feature and tenant, but hard limits need graceful behaviour so a financial control does not corrupt a workflow. For self-hosting, allocate hardware and platform costs using measured utilisation while retaining idle capacity as an explicit reliability choice.

Cost controls belong in the architecture

A production gateway can enforce model allowlists, maximum context and output sizes, concurrency limits and per-feature budgets before requests reach a provider. These controls should return explicit application errors or route to an evaluated fallback, not silently truncate evidence or switch to an unknown model. Detect accidental amplification such as recursive agent calls, repeated retrieval and clients retrying after a timeout while the original request continues. Cancellation should propagate where providers and tools support it, and completed responses that arrive after client disconnection should still be accounted for. Alerts are more useful when they identify the prompt, model and task whose unit cost changed. Financial controls also interact with abuse and security. Public endpoints need authentication, quotas and input limits so an attacker cannot convert arbitrary text into an uncapped invoice. Tool use can incur non-model charges and must be bounded independently. Separate experimentation budgets from production, and prevent development credentials from using unrestricted premium models. When a limit is reached, preserve auditability: record what was rejected and why without retaining unnecessary sensitive content. Review the controls alongside quality metrics. An output cap that systematically cuts off citations or a cheap route that increases manual corrections has moved cost rather than reduced it. Good cost governance constrains waste while keeping product behaviour explicit.

Cost controls belong at the orchestration boundary. Enforce model allowlists, context and output ceilings, concurrency limits and feature budgets before calling a provider. Return an explicit error or evaluated fallback instead of silently truncating evidence. Detect recursive agent calls, repeated retrieval and client retries whose original work continues after timeout. Propagate cancellation where possible and account for late completions. Public endpoints also need authentication and quotas so arbitrary input cannot become an uncapped invoice. Version every prompt and routing policy, allowing finance and engineering to explain changes from the same trace. Review controls against quality because a cheap route that creates manual correction has only moved the expense.

Build the model around completed work

Choose a unit that maps to value: a reviewed document, resolved request, accepted draft or completed investigation. Trace all computation and manual handling needed to reach that outcome. Establish quality and latency thresholds, then compare architectures on cost within those constraints. Revisit the figures whenever prompts, retrieval, provider rates or model versions change. This makes trade-offs legible: whether better retrieval saves generation, whether a smaller model increases repair, or whether reserved capacity suits steady demand. Production inference economics becomes manageable when accounting follows the system that users actually experience, rather than the single endpoint whose price happens to be easiest to quote.