Fine-tuning a small model for one narrow job
When a 1B model with good data beats a frontier model with a clever prompt, plus the dataset, loss curve and eval that convinced me.
Fine-tuning is the third option, not the first
The order is always the same: prompt it, then add retrieval or tools, and only then consider training. Fine-tuning teaches behaviour and format. It does not teach knowledge you can retrieve, and it will not fix a task the model does not understand.
What it is genuinely good at: one narrow job, done thousands of times a day, where consistency matters more than brilliance.
The dataset is the project
A few thousand good examples beat a hundred thousand scraped ones. Good means consistent: the same input style the users will produce, the same output style you want back, including the boring cases and the near misses that a naive labeler would skip.
- Real inputs, not synthetic ones, wherever possible.
- Hard negatives: examples that look right but are wrong.
- A held-out split defined before training starts, and never touched again.
If two humans disagree about a label, the model will too.
LoRA instead of full fine-tuning
Low-rank adapters train a small number of extra parameters, so it fits on modest hardware, runs in minutes to hours, and produces a file measured in megabytes. For format and behaviour changes, the gap to full fine-tuning is usually small enough that you will not care.
Reading the loss curve
A falling training loss with a rising validation loss is memorisation, and it starts earlier than most people expect. Early stopping is not a compromise, it is the correct behaviour. Save checkpoints so you can pick the best one on evals rather than on loss.
The other trap is a curve that never moves: usually a formatting bug in the input template, not a learning rate problem.
Evaluate against the thing you replaced
The baseline is not a random model, it is the prompt-and-retrieval version you were already running. Compare on the same held-out set, on the same machine, with the same scoring.
A small model that is slightly less clever but twice as fast, half the cost, and far more consistent often wins the argument with whoever pays the bill.
