How do you stop an LLM inventing answers?

Treat invention as a system failure

Language models generate likely continuations rather than consulting an internal, authoritative database. Fluent wording and calibrated truth are therefore separate properties. A prompt that says not to hallucinate cannot supply missing evidence or validate a claim. The application must define what sources are authoritative, when an answer is permitted and what happens when support is absent. Different failure classes need different controls: an obsolete policy retrieved correctly is a content-lifecycle problem, while a fabricated value absent from retrieved evidence is a generation or verification problem.

Begin with consequence. A loose draft for internal editing can tolerate errors that a payment instruction, clinical summary or contractual answer cannot. Classify fields and actions by harm, reversibility and review route. Deterministic systems should calculate totals, enforce permissions and validate identifiers. The model can interpret requests and prepare proposals, but it should not become the authority for facts already held in systems of record. This boundary reduces both invention and the subtler risk of a plausible answer overriding a correct structured value.

Ground answers in controlled evidence

Retrieval-augmented generation supplies relevant passages at request time, but only if ingestion, chunking, ranking and permissions work. Store source identity, version, effective date and stable location with each passage. Hybrid retrieval often handles both semantic questions and exact codes better than embeddings alone. Context assembly should remove duplicates and preserve qualifications around a statement. If the required evidence is absent, the correct behaviour is to decline or ask for information. A larger model cannot recover a document that never entered its context.

• Require every material factual claim to map to supplied evidence.
• Reject tool arguments that fail schema, permission or business-rule validation.
• Separate untrusted retrieved content from application instructions.
• Make insufficient evidence a normal, tested response state.
• Log source versions and model configuration for replay and review.

Citations must entail the claim

A citation can be genuine yet irrelevant. Verification should check that the cited passage supports the nearby claim, including dates, scope and exceptions. Generate from numbered evidence units, retain identifiers through context transformations and reject citations that reference material not supplied. For structured extraction, spans can point directly to source offsets. Displaying citations also enables human review, but it does not transfer responsibility to the reader. Automated entailment checks can prioritise review; they are another model judgement and should not be treated as proof.

Constrain actions outside the model

Agents add operational risk because an invented assumption can become a real action. Tools should expose narrow, typed operations rather than a general database or shell. The application authenticates the user, authorises each operation, validates arguments and enforces transaction limits independently. Read operations and proposed changes should be distinguishable from committed changes. High-consequence actions require confirmation that states exactly what will happen. Idempotency keys, audit records and compensating procedures limit damage when calls are repeated or partial failures occur.

Evaluate claims, refusals and edge cases

A representative evaluation set should contain answerable questions, unanswerable questions, conflicting sources, superseded documents and adversarial instructions embedded in content. Score whether material claims are supported, whether citations are correct and whether the system abstains at the intended boundary. Track retrieval separately so generation is not blamed for absent context. Human reviewers need a rubric and examples of acceptable inference. Each change to model, prompt, retrieval or tools runs against the same versioned set before release, with failures examined by category rather than hidden in one average.

Monitor without retaining everything

Production monitoring should capture model and prompt versions, retrieval identifiers, tool outcomes, validation failures and user corrections while minimising sensitive content. Sampling can support quality review where policy permits it. Feedback buttons alone produce biased evidence because many plausible errors pass unnoticed. Operational signals such as repeated retries, citation openings, declined answers and overridden proposals help locate friction but require interpretation. Incident procedures should allow a model route, source collection or tool to be disabled quickly, and corrected content must propagate through indexes and caches. Reviewers should be able to reconstruct the evidence available at the time rather than judging against a source that has since changed.