Quickstart with Ollama

1 Install Ollama

macOS and Windows: grab the installer from ollama.com/download. Linux: one command. Afterwards confirm the version prints in a terminal.

curl -fsSL https://ollama.com/install.sh | sh
ollama -v

2 Run your first model

One command: the first run downloads the weights automatically (about 5 GB for an 8B model), then drops you straight into a chat. Type /bye to exit.

Behind the scenes this does three things: downloads GGUF weights at a quantized size, loads the model into VRAM or system memory, and starts a local inference server — everything afterwards builds on that server.

ollama run qwen3:8b

3 Match the model size to your VRAM

When VRAM runs out, Ollama offloads layers to system memory — it still runs, but noticeably slower. Order-of-magnitude reference for Q4 quants with short contexts: 4B ≈ 3 GB, 8B ≈ 6 GB, 14B ≈ 10 GB, 32B ≈ 20 GB of VRAM.

Real numbers vary a lot with context length, concurrency, and version combinations — which is exactly why this site records the full test environment for every benchmark instead of publishing a bare average.

4 No VRAM headroom, or no discrete GPU

Step down a size: 4B-class models run on integrated graphics and on CPU alone. CPU-only generation is roughly an order of magnitude slower — slow but fully usable, fine for a first taste before deciding on hardware.

ollama run qwen3:4b

5 Use it as a local API

While running, Ollama serves an OpenAI-compatible endpoint: point a client or IDE extension at the local base URL and it just works. Check what is loaded and how much it occupies at any time.

ollama serve
# OpenAI-compatible endpoint: http://localhost:11434/v1
ollama ps

6 Next steps

Want more control — custom context length, picking quantization levels, benchmarking your machine — continue with "Manual setup with llama.cpp". And if you end up with numbers of your own, send them in via the Contact page to grow the database.