NVIDIA RTX 5090
GPUNVIDIA Blackwell flagship (GB202): 32 GB GDDR7 on a 512-bit bus, ~1.79 TB/s of memory bandwidth, 575 W board power.
288
Run records
46
Models covered
8
Frameworks covered
L0 Self-reported × 122 · L1 Reproduced × 1 · L2 Cross-framework verified × 165
Evidence mix
Typical measured performance
Published benchmark averages per model × framework × quantization on this hardware.
- llama.cpp · IQ1_S Decode 381.92 · Prefill 1480.95 ·
- vLLM · Q3_K_S Decode 337.39 · Prefill 1195.07 ·
- vLLM · BF16 Decode 171.53 · Prefill 804.96 ·
- Ollama · Q4_K_M Decode 279.75 · Prefill 1289.8 ·
- Ollama · Q2_K Decode 351.2 · Prefill 1172.76 ·
- llama.cpp · Q4_K_S Decode 219.01 · Prefill 766.77 ·
- Ollama · Q5_K_S Decode 174.79 · Prefill 860.94 ·
- Ollama · Q8_0 Decode 138.97 · Prefill 570.96 ·
- llama.cpp · IQ2_M Decode 200.6 · Prefill 837.93 ·
- Ollama · IQ1_S Decode 346.53 · Prefill 1247.2 ·
- vLLM · IQ1_S Decode 332.71 · Prefill 1515.74 ·
- Strata · Q6_K Decode 310.61 · Prefill 1498.07 ·
- Ollama · IQ3_XS Decode 206.43 · Prefill 827.13 ·
- Strata · Q2_K Decode 376.49 · Prefill 1620.15 ·
- vLLM · Q5_K_S Decode 316.53 · Prefill 1148.64 ·
- Ollama · Q3_K_M Decode 315.13 · Prefill 1204.27 ·
- llama.cpp · IQ2_XS Decode 304.42 · Prefill 1543.97 ·
- llama.cpp · Q3_K_S Decode 289.63 · Prefill 1418.95 ·
- vLLM · Q2_K Decode 312.49 · Prefill 1617.22 ·
- vLLM · Q8_0 Decode 156.33 · Prefill 685.96 ·
- Ollama · IQ3_S Decode 275.39 · Prefill 1404.89 ·
- Strata · IQ4_XS Decode 340.69 · Prefill 1198.68 ·
- Strata · IQ2_M Decode 284.29 · Prefill 1098.95 ·
- llama.cpp · IQ2_XS Decode 262.05 · Prefill 997.54 ·
- llama.cpp · Q8_0 Decode 125.52 · Prefill 751.15 ·
- Ollama · Q3_K_S Decode 236.13 · Prefill 819.72 ·
- Ollama · Q5_K_S Decode 165.12 · Prefill 690.64 ·
- Ollama · Q5_K_S Decode 220.5 · Prefill 854 ·
- Strata · IQ3_S Decode 280 · Prefill 1358.66 ·
- Strata · Q5_K_M Decode 297.96 · Prefill 1229.8 ·
- vLLM · IQ3_XS Decode 353.35 · Prefill 1757.41 ·
- vLLM · Q4_K_M Decode 288.35 · Prefill 1282.34 ·
- Ollama · Q5_K_M Decode 226.28 · Prefill 869.25 ·
- llama.cpp · Q4_K_S Decode 375.58 · Prefill 1439.62 ·
- llama.cpp · Q5_K_S Decode 373.63 · Prefill 1720.71 ·
- vLLM · IQ4_XS Decode 321.22 · Prefill 1224.88 ·
- vLLM · Q6_K Decode 292.37 · Prefill 1232.31 ·
- vLLM · Q8_0 Decode 145.18 · Prefill 478.54 ·
- Ollama · IQ3_XS Decode 161.9 · Prefill 672.48 ·
- Ollama · IQ2_M Decode 234.19 · Prefill 870.04 ·
- llama.cpp · Q3_K_M Decode 377.37 · Prefill 1718.05 ·
- llama.cpp · Q4_K_S Decode 334.66 · Prefill 1225.86 ·
- Strata · Q2_K Decode 338.27 · Prefill 1161 ·
- Strata · Q4_K_M Decode 318.56 · Prefill 1437.9 ·
- llama.cpp · Q2_K Decode 378.36 · Prefill 1748.97 ·
- vLLM · Q3_K_M Decode 330.24 · Prefill 1384.87 ·
- vLLM · Q4_K_S Decode 330.77 · Prefill 1336.98 ·
- Ollama · Q3_K_S Decode 237.4 · Prefill 1150.93 ·
- Strata · Q3_K_L Decode 349.94 · Prefill 1124.59 ·
- llama.cpp · IQ4_XS Decode 256.42 · Prefill 1150.82 ·
Fit check
Physics-based estimate from VRAM and memory bandwidth: pick a model to see which quantizations fit and the expected speed
Formula: tok/s = (bandwidth ÷ model size) × 0.55, adapted from llmfit; see run records for measured numbers
Key specs
- Vendor
- NVIDIA
- Architecture
- Blackwell
- VRAM
- 32 GB
- Memory bandwidth
- 1792 GB/s
- TDP
- 575 W
- Release date
- 2025-01-30
Run records
50 configurations
| Model | Framework | Quantization | Decode | Gen time | Samples |
|---|---|---|---|---|---|
| Ministral-3-8B-Instruct-2512 | llama.cpp | IQ1_S | 381.9 tok/s | — | 1 |
| Ministral-3-8B-Instruct-2512 | vLLM | Q3_K_S | 337.4 tok/s | — | 1 |
| NVIDIA-Nemotron-3.5-Bolt-E4B-4GB | vLLM | BF16 | 171.5 tok/s | — | 1 |
| NVIDIA-Nemotron-3.5-Bolt-E4B-4GB | Ollama | Q4_K_M | 279.8 tok/s | — | 1 |
| Raon-1-mini-preview | Ollama | Q2_K | 351.2 tok/s | — | 1 |
| elephant-1 | llama.cpp | Q4_K_S | 219 tok/s | — | 1 |
| elephant-1 | Ollama | Q5_K_S | 174.8 tok/s | — | 1 |
| elephant-1 | Ollama | Q8_0 | 139 tok/s | — | 1 |
| Qwen3.8-27B | llama.cpp | IQ2_M | 200.6 tok/s | — | 1 |
| Qwen3.6-35B-A3B | Ollama | IQ1_S | 346.5 tok/s | — | 1 |
| Qwen3.5-4B | vLLM | IQ1_S | 332.7 tok/s | — | 1 |
| Qwen3.5-4B | Strata | Q6_K | 310.6 tok/s | — | 1 |
| Qwen3.5-27B | Ollama | IQ3_XS | 206.4 tok/s | — | 1 |
| Qwen3.5-27B | Strata | Q2_K | 376.5 tok/s | — | 2 |
| Qwen3.5-0.8B | vLLM | Q5_K_S | 316.5 tok/s | — | 1 |
| Qwen3.5-0.8B | Ollama | Q3_K_M | 315.1 tok/s | — | 1 |
| Qwen3.8-9B | llama.cpp | IQ2_XS | 304.4 tok/s | — | 1 |
| Qwen3.8-9B | llama.cpp | Q3_K_S | 289.6 tok/s | — | 1 |
| Qwen3.8-9B | vLLM | Q2_K | 312.5 tok/s | — | 1 |
| Qwen3.8-9B | vLLM | Q8_0 | 156.3 tok/s | — | 2 |
| Qwen3-8B | Ollama | IQ3_S | 275.4 tok/s | — | 1 |
| Qwen3-8B | Strata | IQ4_XS | 340.7 tok/s | — | 1 |
| gemma-4-31B-it | Strata | IQ2_M | 284.3 tok/s | — | 1 |
| gemma-4-12B-it | llama.cpp | IQ2_XS | 262.1 tok/s | — | 2 |
| gemma-4-12B-it | llama.cpp | Q8_0 | 125.5 tok/s | — | 2 |
| gemma-4-12B-it | Ollama | Q3_K_S | 236.1 tok/s | — | 1 |
| gemma-4-12B-it | Ollama | Q5_K_S | 165.1 tok/s | — | 1 |
| Llama-3.1-8B-Instruct | Ollama | Q5_K_S | 220.5 tok/s | — | 1 |
| Llama-3.1-8B-Instruct | Strata | IQ3_S | 280 tok/s | — | 1 |
| Llama-3.1-8B-Instruct | Strata | Q5_K_M | 298 tok/s | — | 1 |
| Llama-3-8B-Instruct | vLLM | IQ3_XS | 353.4 tok/s | — | 2 |
| Llama-3-8B-Instruct | vLLM | Q4_K_M | 288.4 tok/s | — | 1 |
| Llama-3-8B-Instruct | Ollama | Q5_K_M | 226.3 tok/s | — | 1 |
| Llama-3.2-3B-Instruct | llama.cpp | Q4_K_S | 375.6 tok/s | — | 2 |
| Llama-3.2-3B-Instruct | llama.cpp | Q5_K_S | 373.6 tok/s | — | 1 |
| Llama-3.2-3B-Instruct | vLLM | IQ4_XS | 321.2 tok/s | — | 1 |
| Llama-3.2-3B-Instruct | vLLM | Q6_K | 292.4 tok/s | — | 1 |
| DeepSeek-R1-Distill-Qwen-14B | vLLM | Q8_0 | 145.2 tok/s | — | 1 |
| Qwen3.8-27B | Ollama | IQ3_XS | 161.9 tok/s | — | 1 |
| Qwen3.6-27B | Ollama | IQ2_M | 234.2 tok/s | — | 1 |
| Qwen3.5-4B | llama.cpp | Q3_K_M | 377.4 tok/s | — | 1 |
| Qwen3.5-4B | llama.cpp | Q4_K_S | 334.7 tok/s | — | 2 |
| Qwen3.5-4B | Strata | Q2_K | 338.3 tok/s | — | 1 |
| Qwen3.5-4B | Strata | Q4_K_M | 318.6 tok/s | — | 1 |
| Qwen3.5-0.8B | llama.cpp | Q2_K | 378.4 tok/s | — | 1 |
| Qwen3.5-0.8B | vLLM | Q3_K_M | 330.2 tok/s | — | 1 |
| Qwen3.5-0.8B | vLLM | Q4_K_S | 330.8 tok/s | — | 1 |
| Qwen3.8-9B | Ollama | Q3_K_S | 237.4 tok/s | — | 1 |
| Qwen3.8-9B | Strata | Q3_K_L | 349.9 tok/s | — | 1 |
| Qwen3-8B | llama.cpp | IQ4_XS | 256.4 tok/s | — | 2 |