What does an AI feature cost to run?

Cost belongs to a workflow

There is no useful universal price for an AI feature. A short classification request, a document assistant with retrieval and an agent executing several tools consume different resources and create different review obligations. Define the business transaction first: its input size, context sources, output requirement, tool sequence, latency target and escalation path. Then measure representative traces. Multiplying one demonstration prompt by projected users misses long documents, conversation history, retries, failed tool calls and the uneven demand that determines provisioned capacity.

Hosted APIs usually charge for input and output units, sometimes with separate prices for cached input, tools or specialised models. Input can dominate when the application repeatedly sends policy text or conversation history; output can dominate long-form generation. Retrieval adds embedding, indexing, vector or lexical queries and document processing. Speech, image and reranking services add their own units. Prices change, so assumptions should be stored as editable inputs with effective dates rather than buried in a proposal or application constant.

Build a request-level cost model

Instrument a complete request and record model route, input and output units, cached units, retrieval queries, reranking, tool calls, retries and elapsed time. Group traces by task rather than averaging all traffic. A simple lookup and a complex analysis should have separate distributions. Estimate normal, busy and exceptional periods, including batch jobs that coincide with interactive demand. For self-hosting, replace per-call price with accelerator, compute, storage, network and operational capacity, then account for idle headroom required to meet latency and resilience objectives.

• Measure context after retrieval and assembly, not only the user message.
• Include unsuccessful requests, retries and fallback model routes.
• Separate one-off ingestion from recurring re-indexing and query costs.
• Cost human review and exception handling as part of the workflow.
• Set budgets and alerts by feature, tenant or task where attribution permits.

Self-hosting changes the cost shape

An open-weight model removes a hosted inference tariff but not inference cost. Hardware must accommodate model memory, concurrency and latency, with redundancy for maintenance and failure. Quantisation and batching can improve utilisation but may alter quality or response time. Engineers must maintain serving software, drivers, security patches, scaling and model artefacts. Self-hosting becomes economically credible under a sustained, predictable workload or a deployment constraint; intermittent traffic often pays for idle capacity. Compare equivalent evaluated quality, not models with merely similar parameter counts.

Optimise architecture before wording

The strongest savings often come from avoiding unnecessary model work. Deterministic code can validate, calculate or route simple cases. Retrieval can select concise evidence instead of sending whole documents. Conversation summaries can cap history growth if their accuracy is tested. A smaller model can handle bounded extraction while difficult cases route to a more capable one. Semantic caching is suitable only when permissions, freshness and input equivalence are enforced. Token trimming without understanding evidence can reduce the bill by removing the qualification that makes an answer correct.

Quality and review are economic variables

A cheaper model that creates more corrections can cost more at workflow level. Measure accepted outputs, abstentions, escalations and rework alongside consumption. Human review may be mandatory for consequence rather than model quality, and its queueing time affects service capacity. Conversely, well-calibrated abstention can be economically preferable to confident errors. Evaluation should establish the least costly route that meets the defined quality boundary for each task. That makes model routing a controlled policy, not an opaque attempt to reduce token expenditure.

Keep forecasts revisable

A forecast should show request distributions, provider prices, exchange assumptions, growth scenarios and operational overhead separately. Validate it during a limited release with budget caps and attribution tags. Reconcile invoices against telemetry; unexplained differences can reveal retries, untracked services or incorrect token accounting. Monitor changes in input length and tool depth as product behaviour evolves. Provider price reductions should not be presumed, and current promotional allowances should not support a durable business case. A model that remains transparent can be updated when workload, architecture or commercial terms change. Procurement commitments should follow observed demand, because reserved capacity bought too early can turn a theoretical discount into avoidable fixed cost.