How to benchmark a local LLM on Mac with llama-bench

A useful local LLM speed test answers a specific question: how fast does this exact model file run with these settings on this Mac? With llama-bench, keep prompt processing and token generation separate, save the raw results, and record enough about the setup to repeat the test.

This walkthrough uses the command from our published Apple M5 benchmark. It ran one checksum-pinned GGUF on a 24 GB MacBook Air using llama.cpp build 10470, commit 34af94cd9. The numbers below are that recorded run, not a promise about a different model or Mac. You need llama-bench installed and a local GGUF already available.

What do prompt processing and token generation mean?

Prompt processing measures work on the input tokens. Token generation measures production of output tokens. Both are reported in tokens per second, but they describe different work.

Our command requested 512 prompt tokens and 128 generated tokens. In its raw JSON, that produced:

Test	n_prompt	n_gen	n_depth	Recorded mean
Prompt processing	512	0	0	39.44 tokens/s
Token generation	0	128	0	1.87 tokens/s

The generation row did not follow the 512-token prompt. Adding those token counts together does not turn this into a 640-token conversation test. If your question is how generation behaves after a long conversation, this particular result does not answer it.

The version-pinned llama-bench documentation shows prompt-processing and generation tests separately, along with the available output formats.

Record the model and Mac before testing

Use the same Terminal session for the commands below. Replace the example model path with your existing GGUF. A new temporary directory keeps each run’s output separate.

benchmark_model='/absolute/path/to/your-model.gguf'
benchmark_dir=$(mktemp -d /tmp/llama-bench-run.XXXXXX)

shasum -a 256 "$benchmark_model" > "$benchmark_dir/model-sha256.txt"
sw_vers > "$benchmark_dir/macos.txt"
sysctl -n machdep.cpu.brand_string > "$benchmark_dir/chip.txt"
sysctl -n hw.memsize > "$benchmark_dir/memory-bytes.txt"
llama-bench --help > "$benchmark_dir/llama-bench-help.txt" 2>&1

Check that each command succeeds before starting the benchmark. The model checksum identifies the bytes you tested; a filename alone is a weaker record. Save the model’s source URL and revision with it so someone else can locate the same artifact.

Also note whether the Mac is plugged in, its power mode, other heavy applications running, and the memory pressure and swap you observe before and during the test. These are recommended controls for a new run. Our historical benchmark did not capture all of them, so we cannot use that record to explain why the machine achieved a particular speed.

Run the recorded llama-bench workload

This is the argument set used in our benchmark, with your local model path substituted. The thread and batch values are reproduction inputs from that Mac, not recommended defaults for every machine. If you change them, label the result as a different configuration.

llama-bench \
  --model "$benchmark_model" \
  --n-prompt 512 \
  --n-gen 128 \
  --repetitions 5 \
  --threads 10 \
  --n-gpu-layers 99 \
  --batch-size 2048 \
  --ubatch-size 512 \
  --flash-attn auto \
  --output json \
  > "$benchmark_dir/results.json" \
  2> "$benchmark_dir/runtime.log"

Wait for completion and check the exit status with echo $? immediately afterward. A nonzero exit status, incomplete JSON, or a load error is a failed run, not a low performance score. Inspect runtime.log before changing settings. You can interrupt the process with Ctrl+C; keep any partial output labelled as incomplete.

The commands write results into the temporary directory. Move that directory to a permanent location when the run is complete; it is not a durable archive. They do not install software or change the model file. Check logs for private paths before sharing them.

Read the raw rows, not just the headline number

In results.json, identify the workload from n_prompt, n_gen, and n_depth. Then retain build_commit, build_number, model information, backend information, and the argument-related fields alongside avg_ts and samples_ts.

Our generation samples were 1.64, 1.54, 1.69, 1.77, and 2.70 tokens/s, rounded to two decimals. Their recorded mean was 1.87 tokens/s. Reporting only the fastest sample would describe a different result from reporting the mean. The variation is visible; its cause is not established by these samples.

The raw benchmark artifact contains our normalized summary and the original llama-bench rows under raw. Your direct --output json file contains the rows themselves, without that site’s wrapper.

How to compare two runs fairly

Start by writing down the change you want to evaluate. If you change the model, runtime, and batch size together, the result cannot isolate which change explains a difference.

For a comparison of runtime settings, preserve the model checksum, runtime build, workload, repetitions, and machine conditions except for the setting under test. Record every changed argument. Repeat both configurations and vary their order rather than always running one first. That is a proposed comparison procedure, not a control we claim for the historical run above.

Use a simple run ledger:

Record	Why keep it?
Model source, revision, and SHA-256	Identify the exact file
Runtime build and full command	Identify the tested implementation and settings
Mac, memory, OS, power, and observed load	Make the machine conditions visible
Workload fields and every sample	Compare the same kind of work without selecting a favourite result
Errors and incomplete runs	Explain exclusions instead of silently dropping them

What this test cannot tell you

Synthetic throughput does not establish answer quality, successful coding tasks, or the full response time of a chat application. Our artifact also does not contain a simultaneous memory trace, so the 24 GB machine specification is not a measured peak-memory requirement for the model.

For choosing a model you can actually work with, combine a repeatable speed test with your own task prompts and a record of failures. The 24 GB Mac model-selection guide shows how a separate selection record used more than one speed number. Its retained model has a different checksum from the benchmark used here; do not merge the two into a single model result.

For the evidence we require when publishing a result, see the benchmark methodology.