When to Fine-Tune an LLM, and When Not To

Fine-tuning is sometimes proposed whenever prompting reaches an awkward stage. That skips the central diagnostic question: what exactly needs to change? An LLM application includes instructions, examples, retrieved knowledge, tools, validation and user interaction. Training the model can improve stable behavioural patterns, but it cannot repair missing source data, unreliable tools or an undefined product requirement. It also creates an asset that must be evaluated, versioned and retrained as base models and policies change. We treat fine-tuning as one engineering option in a wider system, not as the inevitable final step after prompt engineering.

Separate knowledge, behaviour and capability

A knowledge problem asks the model to use facts that are private, current or frequently revised. Retrieval or tools are usually appropriate because they preserve provenance and can reflect an update without training. A behaviour problem concerns format, tone, classification boundaries or a repeated transformation. Fine-tuning can help when that behaviour is demonstrated consistently across many examples. A capability problem means the model cannot reliably reason over the task even when given good context and instructions; a more capable base model, decomposition or deterministic algorithm may be required. These categories overlap, but naming the dominant one avoids trying to teach annual prices into weights or using a vector index to enforce a strict output grammar.

Exhaust cheaper controls for the right reasons

Prompting is appropriate when requirements can be stated compactly, and a few examples often clarify edge cases. Structured output constraints and application validation are better than training for syntax alone. Retrieval supplies inspectable source material. Tool calls move arithmetic, search and state changes into systems designed for them. Workflow decomposition can turn one unstable request into classification, lookup and drafting stages with separate checks. These are not merely prototypes before “real” training; they remain necessary controls after fine-tuning. The goal is not to avoid training at all costs, but to establish a strong baseline and evidence that the remaining errors are learnable behavioural patterns rather than faults elsewhere.

• Consider fine-tuning when a high-volume task has a stable input-output contract.
• Require a representative set of consistently reviewed examples, not synthetic volume alone.
• Prefer retrieval for changing facts, citations and tenant-specific knowledge.
• Prefer deterministic code for permissions, calculations and non-negotiable business rules.
• Use a stronger base model as a benchmark before investing in a training pipeline.

Training data is the product specification

A fine-tuned model learns contradictions and shortcuts as readily as intended behaviour. Examples should come from the target distribution, be stripped or governed for sensitive information and be reviewed against an explicit rubric. Near-duplicate samples can make a validation split look reassuring while adding little generalisation. Split related conversations or source documents together to prevent leakage between train and test sets. Include difficult negative examples where the desired action is refusal, clarification or no tool call. Data quantity matters less than coverage and consistency; disagreement between reviewers often reveals that the task itself lacks a settled answer. Resolve that ambiguity before encoding it in model weights.

Choose the adaptation method deliberately

Supervised fine-tuning teaches response patterns from demonstrations. Parameter-efficient methods such as low-rank adaptation can reduce training and storage requirements when the platform permits them, but they do not remove the need for evaluation. Preference optimisation can shape choices between candidate responses, though preference labels must represent the actual product objective rather than superficial style. Distillation may transfer behaviour from a larger model to a smaller one, trading some capability for lower serving cost. The available techniques and controls depend on whether the model is hosted or self-managed. In every case, retain the base-model version, dataset version, training configuration and evaluation results so the artefact has a traceable lineage.

Evaluate against prompt and model baselines

The tuned model should be compared on a held-out set with the best practical prompted baseline, not with the first prompt somebody wrote. Score task correctness, severe failures, calibration, latency and total serving cost. Check general capabilities that could regress, particularly instruction following, safety behaviour and languages represented sparsely in training. Human review should be blind to candidate identity. Run adversarial and out-of-distribution cases because a narrower model may become confidently brittle outside its training pattern. If the gain appears only on familiar templates or disappears when system instructions change, the training set may have taught surface form rather than the intended task.

Include lifecycle cost in the decision

Training expenditure is only one line. Data collection and review, experiments, safety testing, model hosting, observability and retraining all consume effort. A tuned smaller model may lower per-request inference cost at sufficient volume, but that case depends on measured traffic, token lengths and platform pricing. Base-model upgrades may require repeating adaptation and regression tests. Data deletion obligations can be harder to satisfy for information incorporated into weights than for indexed documents. Teams also need a rollback route and a way to route unsupported cases to the baseline. The business case should compare lifetime cost per successful task, including review and failure handling, rather than cost per training run.

Know what the experiment can prove

A small adaptation trial can answer whether a method learns a target pattern and whether the serving platform is workable. It cannot establish performance on traffic absent from the dataset. Define the hypothesis narrowly, such as improving correct classification of a known set of ambiguous intents without increasing critical false positives. Freeze the evaluation set and baseline before training, then run more than one seed or data sample where training variance could affect the conclusion. Review learning curves for memorisation and inspect errors rather than selecting a checkpoint solely by aggregate validation loss. Keep examples grouped by source to prevent related records leaking across splits. Plan deployment before declaring the experiment successful. The tuned model still needs system instructions, safety controls, runtime input validation and observability. Decide how requests are routed, how the model identifier is recorded in traces and how rollback works when a provider endpoint or adapter changes. Monitor drift in input classes and reviewer corrections; a stable model can deteriorate relative to a changing task. Establish triggers for refreshing data or abandoning the tuned variant. A disciplined experiment produces more than weights: it produces a dataset lineage, a measured comparison, known unsupported cases and an operational plan. Without those artefacts, an apparent quality gain is difficult to trust or maintain.

A small experiment should test a narrow hypothesis against a frozen baseline and held-out set. Group related records by source to prevent leakage, inspect learning curves and error classes, and use the decoding settings intended for production. Where training variation matters, compare more than one run rather than reporting the most favourable checkpoint. Plan deployment before accepting the result: routing, model identifiers in traces, rollback and monitoring for input drift all affect whether the gain can be maintained. Record failed experiments as well as the selected artefact. Useful output from a trial includes dataset lineage, unsupported cases and an operating plan, not just weights and a validation score.

Make the decision from evidence

Start with an error analysis of real or representative traffic. Label each failure as missing knowledge, instruction ambiguity, formatting, tool use, reasoning or policy. Build the strongest non-training baseline and measure it. If a substantial, important residue consists of stable behaviours that good demonstrations can express, prepare a governed dataset and run a small experiment. Set acceptance criteria before seeing results, compare against baselines and test outside the training distribution. Fine-tune when the measured improvement justifies the ongoing model lifecycle. Otherwise keep the system simpler, and invest in the retrieval, workflow or product specification that the diagnosis actually identified.