Local LLMUpdates
Search

Guides · published

How to run a local LLM on a 24GB Mac with llama.cpp

Start a local GGUF with llama-server, open the browser chat, check a first response, and stop the server. Includes the exact setup tested on one 24GB M5 Mac.

Published September 5, 2026 · Local LLM Updates Editorial

To run a local LLM with llama.cpp, you need two things: the runtime and a compatible model file. llama-server loads that file and provides a browser chat. This walkthrough starts it on a loopback address, so the command does not bind the server to your Wi-Fi or Ethernet address.

We tested the startup command and a short response on one Apple M5 Mac with 24 GB of unified memory. The runtime and model were already installed. This is a tested startup procedure, not a clean-Mac installation test or a claim that every 27B model runs comfortably within 24 GB.

1. Check the runtime

Open Terminal and check whether the command is available:

llama-server --version

Our tested copy reported build 10470, commit 34af94cd9, installed through Homebrew. If you already have a working version, record it before changing anything.

If you have Homebrew but do not have llama.cpp, the official formula gives this installation command:

brew install llama.cpp

Then run llama-server --version again. Package-manager installs can deliver a different build from the one tested here. Use your installed llama-server --help to check the options below. We did not reinstall Homebrew or llama.cpp for this startup test.

2. Choose the model file before starting the server

Use a GGUF whose source, revision, and size you can identify. Downloading a file and loading it are separate steps; the startup command below uses an existing local file and does not request a model from a remote repository.

For this smoke test, we reused the model from the 24 GB Mac selection record:

  • Repository: huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF.
  • Revision: 6a97e23be66dffc92c2d4e04b17c6daca29a63d5.
  • File: Huihui-Qwen3.8-27B-abliterated-UD-Q2_K_XL.gguf.
  • File size: 10,011,909,024 bytes.
  • SHA-256: 3715d3c31491d706c164445246d31f7ac59822a735d3ae53560b826e248b0e73.

That identifies the file we tested; it is not a recommendation to start with a 27B model on every Mac. Our working machine had background applications and substantial swap. A successful short response does not establish a comfortable memory margin for sustained use.

Set a variable to your actual file path. Keep the quotes if the path contains spaces:

local_model='/absolute/path/to/your-model.gguf'
shasum -a 256 "$local_model"

The checksum above is relevant only if you are using that exact artifact. A different model file should be checked against its own source, not against our checksum.

If you do not have a GGUF yet

The Hugging Face download guide explains the supported download tools. If you already have the hf command, this example requests only the recorded file at its recorded revision, rather than the whole model repository:

hf download huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF \
  Huihui-Qwen3.8-27B-abliterated-UD-Q2_K_XL.gguf \
  --revision 6a97e23be66dffc92c2d4e04b17c6daca29a63d5 \
  --local-dir ./models/mac-guide

We checked these arguments against installed CLI help but did not repeat the download. This example is for reproducing the recorded file, not a recommendation of the model for every Mac. Wait for a successful download, set local_model to the downloaded file’s path, and run the checksum check above before starting the server. If hf is unavailable, follow the installation link in the official download guide first.

3. Start one local chat slot

In the same Terminal window, run:

llama-server \
  --model "$local_model" \
  --alias mac-guide-test \
  --host 127.0.0.1 \
  --port 18765 \
  --ctx-size 2048 \
  --parallel 1 \
  --cache-ram 0 \
  --reasoning off

These are the settings used in our text-only startup test. The model alias gives the API a predictable name. The context is explicitly set to 2,048 tokens, with one server slot. The RAM prompt caching is disabled and reasoning is switched off for this short first-response check. The version-pinned server documentation defines these options. Disabling RAM prompt caching does not remove the context’s KV cache.

Port 18765 kept the test separate from another server already running on the test machine. If it is occupied on your Mac, use another free port and use that same number in every URL below. Do not stop an unfamiliar process just to free a port.

Wait until the log reports that the model is loaded and the server is listening. Leave this Terminal window open. This command does not include a vision projector or an MTP configuration; it is simpler than the separate retained profile in the selection guide.

4. Open the browser chat

Open http://127.0.0.1:18765/ in your browser. Our Chrome check loaded the chat interface and showed the model alias mac-guide-test.

Enter this short test prompt:

Reply with exactly: LOCAL_OK

Our browser test returned LOCAL_OK. This establishes that the tested startup path can produce a text response. It does not measure reasoning quality, coding ability, or general reliability.

For an API check, keep the server running and open a second Terminal window:

curl --fail-with-body http://127.0.0.1:18765/health

Our readiness check returned HTTP 200 and {"status":"ok"}. To send the same short prompt:

curl --fail-with-body http://127.0.0.1:18765/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"mac-guide-test","messages":[{"role":"user","content":"Reply with exactly: LOCAL_OK"}],"max_tokens":16,"temperature":0,"stream":false}'

Our API check also returned LOCAL_OK in choices[0].message.content. If you changed the alias in the server command, use the matching model name in the request.

Common first-run problems

What you see What to check next
llama-server is not found Check that the runtime installation completed and that the command is on your shell’s PATH.
The model file cannot be opened Check the file path, surrounding quotes, download completion, and checksum.
The port cannot be bound Choose a free port and update the browser and API URLs to match.
The browser cannot connect Check that the server is still running and that its log reports readiness.
An option is rejected Compare the command with llama-server --help for the build actually installed.
The Mac becomes unresponsive or swap grows Stop the test and reassess the model and available memory before increasing context or workload.

Stop the server and save the working setup

Return to the Terminal window running llama-server and press Ctrl+C. The command runs in the foreground; stopping it ends that server process without deleting your model file. Our test-owned server was stopped afterward, while the existing server on the other port was left running.

Save the model source and checksum, runtime version, and full startup command. Then try a small set of your own real tasks before deciding whether to keep the configuration.

When the startup path works, the Mac llama-bench guide explains how to measure speed with repeatable workloads. Keep that evaluation separate from this first-response test: two successful short replies are not a benchmark.

The startup evidence records the command, model checksum, responses, observed swap and cleanup, including a failed initial HTTP-client check.