Local LLMs: how far can your own hardware go?
Quantisation, context budgets and small-model tradeoffs, and the point where running a model on your laptop beats paying per token.
Do the memory maths first
The rule of thumb: parameters times bytes per parameter, plus room for the context. A 7B model at 4-bit weights is roughly 4 to 5 GB, and the KV cache grows with context length, so long conversations quietly double your footprint.
If the model does not fit in memory, it runs from disk and the experience collapses. Fitting is the first and most important constraint.
Quantisation, briefly
4-bit is the sweet spot for most work: a small quality hit, a large memory win, everything fits on a laptop or a single consumer GPU. 8-bit is close to lossless and still practical. Below 4-bit, quality falls off a cliff fast, and the cliff arrives at different points for different model families, so test your own task instead of trusting a table.
What small models are good at
- Classification, extraction, and routing: often indistinguishable from a frontier model.
- Summarising a document you already have, in a fixed format.
- Draft work where you are editing anyway.
- Anything private enough that you do not want it leaving the machine.
Where they fall apart
Long multi-step reasoning, subtle writing, and large context. If your task depends on holding twelve documents in mind at once, a small local model will disappoint you, and no amount of prompt engineering closes that gap.
When local actually wins
Not just cost per token. Privacy, latency floor, offline availability, and iteration speed: no rate limits, no SDK churn, and you can run a hundred experiments on a Tuesday without thinking about the bill.
A reasonable split I use: local models for the high-volume, low-stakes steps, an API model for the step a human reads.
The practical stack
llama.cpp or a wrapper around it, a quantised GGUF file, and a smaller model than you think you need. Then measure tokens per second and time to first token on your own hardware, because those two numbers decide more about your users' patience than the leaderboard does.
