How to Evaluate an LLM Feature Before Shipping It

LLM features fail differently from deterministic software. The same prompt can produce several acceptable outputs, a plausible response can contain one consequential error, and a model upgrade can improve the average while breaking an important minority case. Conventional unit tests still cover parsers, permissions and tool adapters, but they cannot establish whether generated work is useful and safe. Before release, the team needs an evaluation contract: what task the feature performs, which mistakes matter, what evidence supports a judgement and what operating limits are acceptable. That contract makes quality a property of the system rather than an opinion formed from a handful of polished examples.

Turn the product promise into a rubric

“Summarise this case” is too vague to test. A useful specification names required facts, forbidden disclosures, acceptable omissions, tone constraints and the action a user takes next. Quality dimensions should be scored separately because a single pass/fail label gives little diagnostic value. For a support draft, reviewers might assess factual support, instruction adherence, completeness and inappropriate commitment. For extraction, field-level precision and recall are more useful than stylistic grading. Define severe failures independently of average quality: leaking another tenant’s data, inventing a financial amount or executing an unauthorised tool call should block release even if most samples are excellent. Rubrics need concrete anchors showing what poor, adequate and strong outputs look like.

Build the dataset from the intended traffic

Evaluation inputs should reflect production distribution without reproducing sensitive data carelessly. Include common cases, long-tail formats, malformed inputs, multiple languages where supported, empty context and deliberate attempts to override instructions. Keep a curated core set for regression and a rotating sample that guards against tuning to a static benchmark. Each example needs source material, expected properties and relevant metadata such as task type and risk tier. Exact reference answers are appropriate for classification or extraction; open generation is better assessed through claims and rubrics. Split dataset creation from prompt tuning where possible, because authors who know every test case can accidentally optimise for the test rather than the product.

• Routine examples matching the most frequent user journeys.
• Boundary cases involving length, formatting, missing fields and conflicting evidence.
• High-impact cases where a subtle error changes a decision or transaction.
• Adversarial instructions, indirect prompt injection and unexpected tool responses.
• Negative cases where the correct behaviour is to ask, refuse or abstain.

Use several graders and calibrate them

Programmatic checks are reliable for JSON validity, required citations, forbidden strings, numeric consistency and tool-call schemas. Model-based graders can scale judgements about relevance or style, but they inherit biases, can be sensitive to answer order and may reward confident verbosity. Human reviewers remain necessary for ambiguous and high-risk dimensions. Calibrate all graders on the same sample, investigate disagreement and revise rubric language before expanding the run. Blind reviewers to model identity when comparing candidates. If an automated grader is used as a release gate, version its prompt and model just like the system under test, and periodically compare its decisions with human judgement. A grader score without demonstrated agreement is another model output, not ground truth.

Evaluate the whole system, not a transcript

Production behaviour includes retrieval, prompt construction, policy checks, tools, retries and post-processing. A model may choose the correct tool but pass an invalid date, or receive correct search evidence that is truncated before generation. Capture traces at each boundary and score both intermediate decisions and final outcomes. Tool tests should run against controlled fixtures so they are deterministic and cannot mutate live systems. Verify permissions outside the model: the model proposes an action, while application code authenticates, authorises, validates and applies idempotency rules. Test streaming interruptions, timeouts, malformed provider responses and fallback paths. Reliability under these conditions matters as much as response quality on a clean request.

Set release thresholds by risk and segment

One aggregate score can conceal a collapse in a small but important group. Report dimensions by language, input length, workflow, customer configuration and risk class. Establish a baseline from the current production path or a simple non-LLM alternative, then require the candidate to meet explicit gates. Some gates are hard constraints, such as no critical permission failures in the test suite. Others permit trade-offs between quality, latency and cost. Include confidence intervals when sample sizes are small, and inspect changed examples rather than celebrating a fractional average gain. The release decision should document accepted weaknesses and the product controls that contain them, such as mandatory review before sending.

Measure latency and cost under load

Offline quality testing says little about queueing, provider limits or the cost of verbose contexts. Measure time to first token and completion latency separately, using realistic input and output lengths. Account for retries, safety calls, retrieval, reranking and tool loops rather than quoting the model endpoint alone. Load tests should respect provider policies while exploring concurrency limits and degraded dependencies. Record token and request costs per successful task, not merely per call, because a workflow that retries twice has a different economic profile. Where caching is allowed, evaluate hit rates with realistic repetition and ensure cache keys include all security and version dimensions that affect the answer.

Test change, not just absolute quality

Most release decisions compare a candidate with an existing prompt, model or non-LLM workflow. Use paired evaluation: run both systems on the same examples and inspect where their outcomes differ. Pairing reduces noise from dataset composition and directs reviewers to changed behaviour. Categorise wins, neutral changes and regressions by rubric dimension and risk. A candidate may write better prose while losing a required citation, or improve English requests while mishandling another supported language. Store the complete candidate traces so reviewers can identify whether the difference came from retrieval, tool selection or generation. Do not repeatedly modify prompts after viewing the held-out set; reserve an untouched final set or collect a fresh sample for the release decision. Evaluation itself needs maintenance. Add a case when production reveals a new failure class, but avoid filling the suite with near-duplicates of every incident. Retire cases only with an explicit reason, and keep critical historical failures as permanent regression tests. Monitor score distributions and grader disagreement over time because provider or grader changes can shift the measuring instrument. When the product workflow changes, revisit the rubric rather than carrying forward a target that no longer represents user value. The result is a living quality system: stable enough to compare releases, yet responsive to evidence from real use.

Release as an observable experiment

After offline gates pass, use staged exposure, feature flags and a defined rollback path. Monitor user corrections, abandonment, tool failures, refusals, latency and spend, but interpret behavioural signals carefully: acceptance does not guarantee correctness. Sample production traces for structured review under an appropriate data policy and feed newly discovered cases back into the rotating set. Version prompts, models, retrieval indexes and graders so a result can be reproduced. The actionable sequence is straightforward: specify the job, enumerate serious harm, construct representative cases, calibrate grading, test the entire trace, set segmented gates and observe a controlled release. That is less dramatic than a demo, and far more useful.