A useful benchmark combines machine performance with task quality. Tokens per second alone cannot tell you whether the model is worth using.
Why internet benchmark tables are not enough
Performance changes with the exact model file, quantization, runtime, context, operating system, drivers, CPU, GPU offload and thermal limits. Results from another machine are useful clues, not promises.
LocalLens therefore avoids predicted speed. Benchmark the shortlisted model on your actual device with the settings and job you intend to use.
Record the test environment
- Device and operating system.
- CPU, total RAM, GPU and VRAM or unified memory.
- Runtime and version.
- Exact model tag or file, quantization and size.
- Context setting, GPU offload, thread count and temperature.
- Whether this is a cold start or the model is already loaded.
Measure four things
First-token latency measures how long the system feels unresponsive. Generation throughput measures output tokens per second. Peak memory reveals whether the setup leaves safe headroom. Task quality measures whether the result is correct and useful.
llama.cpp provides llama-bench and can output structured benchmark results. Its reported prompt-processing and token-generation measurements are valuable, but they still need a task evaluation beside them.
Never compare two models using different prompts, contexts or generation lengths.
Create a 20-case task set
Use examples from the real workflow with sensitive details removed. Define the expected answer or a clear scoring rule before running the models. Include easy, difficult, ambiguous and should-refuse cases.
Score correctness, format following, unsupported claims and completion time. Repeat unstable cases. The winner is the smallest model that clears the required quality threshold at an acceptable latency.
Use a decision scorecard
- Quality pass rate.
- Median time to first token.
- Median complete-response time.
- Peak system and GPU memory.
- Failure and refusal behavior.
- Operational effort: download, startup, updates and integration.
When to retest
Retest after changing the model file, quantization, runtime version, context, prompt template, driver, hardware or workflow. Keep the old results so an upgrade must demonstrate a measurable improvement instead of relying on novelty.
For business deployments, add user acceptance and security tests before release. Deviceterra can help turn the scorecard into a controlled pilot and deployment decision.
Find a model your computer can run.
The LocalLens advisor applies conservative memory rules and tells you when the evidence is insufficient.
Run the free compatibility check →This guide is educational and reviewed for practical accuracy. Model software, licenses and hardware support change; verify official sources before a production deployment.