NVIDIA RTX 5070
GPUNVIDIA Blackwell (GB205): 12 GB GDDR7 on a 192-bit bus, 672 GB/s of memory bandwidth, 250 W board power. Bandwidth is up about a third over the RTX 4070, though the 12 GB capacity bounds model size.
110
Run records
24
Models covered
5
Frameworks covered
L0 Self-reported × 58 · L1 Reproduced × 1 · L2 Cross-framework verified × 51
Evidence mix
Typical measured performance
Published benchmark averages per model × framework × quantization on this hardware.
- Ollama · Q3_K_S Decode 115.6 · Prefill 548.24 ·
- Ollama · Q5_K_S Decode 83.31 · Prefill 418.84 ·
- llama.cpp · Q4_K_M Decode 186.5 · Prefill 627.46 ·
- Ollama · FP16 Decode 84.08 · Prefill 290.65 ·
- Ollama · IQ1_S Decode 239.86 · Prefill 1174.67 ·
- vLLM · IQ1_S Decode 119.3 · Prefill 428.02 ·
- Strata · IQ1_S Decode 153.3 · Prefill 756.44 ·
- Strata · IQ2_XS Decode 135.12 · Prefill 577.01 ·
- llama.cpp · IQ1_S Decode 116.33 · Prefill 578.18 ·
- llama.cpp · Q3_K_M Decode 92.37 · Prefill 403.56 ·
- vLLM · IQ1_S Decode 135.82 · Prefill 714.11 ·
- vLLM · Q2_K Decode 123.84 · Prefill 585.88 ·
- llama.cpp · IQ4_XS Decode 238.16 · Prefill 793.59 ·
- llama.cpp · Q5_K_S Decode 193.78 · Prefill 821.91 ·
- Ollama · FP16 Decode 100.33 · Prefill 407.38 ·
- Ollama · Q3_K_L Decode 203.4 · Prefill 838.81 ·
- vLLM · Q3_K_L Decode 76.49 · Prefill 291.62 ·
- Strata · Q5_K_M Decode 76.48 · Prefill 397.87 ·
- vLLM · Q3_K_M Decode 112.72 · Prefill 436.5 ·
- vLLM · Q4_K_M Decode 84.51 · Prefill 408.98 ·
- Strata · IQ3_S Decode 154.26 · Prefill 563.04 ·
- vLLM · IQ3_XS Decode 223.3 · Prefill 1047.53 ·
- vLLM · IQ4_XS Decode 188.25 · Prefill 873.82 ·
- Ollama · Q3_K_S Decode 197.53 · Prefill 685.98 ·
- Ollama · Q5_K_S Decode 124.72 · Prefill 677.51 ·
- Strata · IQ4_XS Decode 233.02 · Prefill 915.89 ·
- Strata · Q6_K Decode 189.01 · Prefill 746.35 ·
- Ollama · IQ1_S Decode 93.91 · Prefill 396.52 ·
- llama.cpp · IQ2_XS Decode 93.24 · Prefill 441.9 ·
- llama.cpp · Q3_K_S Decode 83.47 · Prefill 327.27 ·
- Strata · IQ3_XS Decode 128.34 · Prefill 480.77 ·
- vLLM · IQ2_M Decode 185.93 · Prefill 658.25 ·
- Ollama · Q3_K_L Decode 127.9 · Prefill 682.85 ·
- Ollama · Q5_K_M Decode 93.03 · Prefill 365.14 ·
- Strata · Q3_K_L Decode 187.58 · Prefill 667.52 ·
- llama.cpp · IQ4_XS Decode 54 · Prefill 640 ·
- llama.cpp · Q4_K_M Decode 50 · Prefill 586 ·
- llama.cpp · Q5_K_M Decode 52 · Prefill 454 ·
- llama.cpp · Q8_0 Decode 36 · Prefill 394 ·
- vLLM · IQ4_XS Decode 51 · Prefill 486 ·
- vLLM · Q4_K_M Decode 47 · Prefill 429 ·
- vLLM · Q5_K_M Decode 42 · Prefill 447 ·
- vLLM · Q8_0 Decode 28 · Prefill 334 ·
- Ollama · IQ4_XS Decode 55 · Prefill 549 ·
- Ollama · Q4_K_M Decode 50 · Prefill 516 ·
- Ollama · Q5_K_M Decode 42 · Prefill 504 ·
- Ollama · Q8_0 Decode 36 · Prefill 356 ·
- Strata · IQ4_XS Decode 70 · Prefill 837 ·
- Strata · Q4_K_M Decode 64 · Prefill 768 ·
- Strata · Q5_K_M Decode 72 · Prefill 739 ·
Fit check
Physics-based estimate from VRAM and memory bandwidth: pick a model to see which quantizations fit and the expected speed
Formula: tok/s = (bandwidth ÷ model size) × 0.55, adapted from llmfit; see run records for measured numbers
Key specs
- Vendor
- NVIDIA
- Architecture
- Blackwell
- VRAM
- 12 GB
- Memory bandwidth
- 672 GB/s
- TDP
- 250 W
- Release date
- 2025-03-05
Run records
50 configurations
| Model | Framework | Quantization | Decode | Gen time | Samples |
|---|---|---|---|---|---|
| Qwen3.5-4B | Ollama | Q3_K_S | 115.6 tok/s | — | 1 |
| Qwen3.5-4B | Ollama | Q5_K_S | 83.3 tok/s | — | 1 |
| Qwen3.5-0.8B | llama.cpp | Q4_K_M | 186.5 tok/s | — | 1 |
| Qwen3.5-0.8B | Ollama | FP16 | 84.1 tok/s | — | 1 |
| Qwen3.5-0.8B | Ollama | IQ1_S | 239.9 tok/s | — | 1 |
| Qwen3.8-9B | vLLM | IQ1_S | 119.3 tok/s | — | 1 |
| Qwen3.8-9B | Strata | IQ1_S | 153.3 tok/s | — | 1 |
| Qwen3.8-9B | Strata | IQ2_XS | 135.1 tok/s | — | 1 |
| Qwen3-8B | llama.cpp | IQ1_S | 116.3 tok/s | — | 1 |
| Qwen3-8B | llama.cpp | Q3_K_M | 92.4 tok/s | — | 1 |
| Llama-3.1-8B-Instruct | vLLM | IQ1_S | 135.8 tok/s | — | 1 |
| Llama-3-8B-Instruct | vLLM | Q2_K | 123.8 tok/s | — | 1 |
| Llama-3.2-3B-Instruct | llama.cpp | IQ4_XS | 238.2 tok/s | — | 1 |
| Llama-3.2-3B-Instruct | llama.cpp | Q5_K_S | 193.8 tok/s | — | 1 |
| Llama-3.2-3B-Instruct | Ollama | FP16 | 100.3 tok/s | — | 1 |
| Llama-3.2-3B-Instruct | Ollama | Q3_K_L | 203.4 tok/s | — | 1 |
| Kimi-K3 | vLLM | Q3_K_L | 76.5 tok/s | — | 1 |
| Kimi-K3 | Strata | Q5_K_M | 76.5 tok/s | — | 1 |
| Qwen3.5-4B | vLLM | Q3_K_M | 112.7 tok/s | — | 1 |
| Qwen3.5-4B | vLLM | Q4_K_M | 84.5 tok/s | — | 1 |
| Qwen3.5-4B | Strata | IQ3_S | 154.3 tok/s | — | 1 |
| Qwen3.5-0.8B | vLLM | IQ3_XS | 223.3 tok/s | — | 1 |
| Qwen3.5-0.8B | vLLM | IQ4_XS | 188.3 tok/s | — | 1 |
| Qwen3.5-0.8B | Ollama | Q3_K_S | 197.5 tok/s | — | 1 |
| Qwen3.5-0.8B | Ollama | Q5_K_S | 124.7 tok/s | — | 1 |
| Qwen3.5-0.8B | Strata | IQ4_XS | 233 tok/s | — | 1 |
| Qwen3.5-0.8B | Strata | Q6_K | 189 tok/s | — | 1 |
| Qwen3.8-9B | Ollama | IQ1_S | 93.9 tok/s | — | 1 |
| Qwen3-8B | llama.cpp | IQ2_XS | 93.2 tok/s | — | 1 |
| Qwen3-8B | llama.cpp | Q3_K_S | 83.5 tok/s | — | 1 |
| Qwen3-8B | Strata | IQ3_XS | 128.3 tok/s | — | 1 |
| Qwen3-1.7B | vLLM | IQ2_M | 185.9 tok/s | — | 1 |
| Qwen3-1.7B | Ollama | Q3_K_L | 127.9 tok/s | — | 1 |
| Qwen3-1.7B | Ollama | Q5_K_M | 93 tok/s | — | 1 |
| Qwen3-1.7B | Strata | Q3_K_L | 187.6 tok/s | — | 1 |
| Qwen3.5-27B | llama.cpp | IQ4_XS | 54 tok/s | — | 1 |
| Qwen3.5-27B | llama.cpp | Q4_K_M | 50 tok/s | — | 1 |
| Qwen3.5-27B | llama.cpp | Q5_K_M | 52 tok/s | — | 1 |
| Qwen3.5-27B | llama.cpp | Q8_0 | 36 tok/s | — | 1 |
| Qwen3.5-27B | vLLM | IQ4_XS | 51 tok/s | — | 1 |
| Qwen3.5-27B | vLLM | Q4_K_M | 47 tok/s | — | 1 |
| Qwen3.5-27B | vLLM | Q5_K_M | 42 tok/s | — | 1 |
| Qwen3.5-27B | vLLM | Q8_0 | 28 tok/s | — | 1 |
| Qwen3.5-27B | Ollama | IQ4_XS | 55 tok/s | — | 1 |
| Qwen3.5-27B | Ollama | Q4_K_M | 50 tok/s | — | 1 |
| Qwen3.5-27B | Ollama | Q5_K_M | 42 tok/s | — | 1 |
| Qwen3.5-27B | Ollama | Q8_0 | 36 tok/s | — | 1 |
| Qwen3.5-27B | Strata | IQ4_XS | 70 tok/s | — | 1 |
| Qwen3.5-27B | Strata | Q4_K_M | 64 tok/s | — | 1 |
| Qwen3.5-27B | Strata | Q5_K_M | 72 tok/s | — | 1 |