Manual setup with llama.cpp
Download GGUF weights, pick your own quantization and flags, and understand every step — more control, higher performance ceiling.
Manual setup with llama.cpp
Get llama.cpp
Download a prebuilt package for your platform from GitHub Releases — it includes llama-server, llama-bench, and the rest of the toolset. For a specific GPU backend or the latest features, build from source following the official docs.
# https://github.com/ggml-org/llama.cpp/releasesDownload GGUF weights and pick a quant
Search "model name + GGUF" on Hugging Face for community-converted weights. Choosing a level: Q4_K_M is the general-purpose balance (8B ≈ 4.7 GB); Q8_0 stays closer to the original model at nearly double the size. Model pages usually include a per-quant size and quality table.
Start the server
llama-server starts a local server with a built-in web chat: -ngl sets how many layers go to the GPU (99 = all; lower it, e.g. 24, if VRAM runs out) and -c sets context length. Open localhost:8080 to chat.
llama-server -m Qwen3-8B-Q4_K_M.gguf -ngl 99 -c 8192
# Web chat: http://localhost:8080The two flags that matter most for speed
GPU offload (-ngl) decides where compute happens: layers that do not fit fall back to the CPU and speed drops off a cliff. Context length (-c) decides KV cache usage: VRAM grows linearly with length, and pushing it too far squeezes out the model itself. Start short and walk up to your machine’s limit.
Public benchmarks often omit the context length they tested at — this site lists it first among "known limitations". Worth verifying explicitly on your own machine.
Benchmark your machine
llama-bench reports two throughputs: pp (prompt processing — this site’s Prefill) and tg (text generation — this site’s Decode). Run it per quant level, per -ngl, per -c setting and you have your machine’s real envelope.
llama-bench -m Qwen3-8B-Q4_K_M.gguf -ngl 99Send the results back
Note the model, hardware, framework version, quantization, and your pp/tg numbers, then submit them via the Contact page — verified public benchmarks with cited sources enter the database, so the next person searching "32B on a 4090" starts from data.