NVIDIA RTX 5070 Ti
GPUNVIDIA Blackwell (GB203): 16 GB GDDR7 on a 256-bit bus, 896 GB/s of memory bandwidth, 300 W board power. Shares the GB203 die with the RTX 5080 but with fewer cores enabled.
111
Run records
25
Models covered
8
Frameworks covered
L0 Self-reported × 86 · L2 Cross-framework verified × 25
Evidence mix
Typical measured performance
Published benchmark averages per model × framework × quantization on this hardware.
- llama.cpp · IQ1_S Decode 182.74 · Prefill 910.54 ·
- vLLM · IQ4_XS Decode 131.32 · Prefill 455.29 ·
- vLLM · Q4_K_M Decode 116.07 · Prefill 492.31 ·
- vLLM · FP16 Decode 121.99 · Prefill 561.36 ·
- vLLM · Q6_K Decode 174.9 · Prefill 685.31 ·
- Strata · Q5_K_M Decode 251.18 · Prefill 1222.13 ·
- Strata · Q6_K Decode 237.15 · Prefill 1031.12 ·
- vLLM · IQ3_S Decode 103.59 · Prefill 387.95 ·
- vLLM · Q3_K_M Decode 98.53 · Prefill 475.97 ·
- Ollama · IQ3_XS Decode 89.42 · Prefill 466.92 ·
- Ollama · Q2_K Decode 98.9 · Prefill 326.08 ·
- llama.cpp · Q3_K_M Decode 110.54 · Prefill 601.05 ·
- Ollama · Q4_K_S Decode 82.54 · Prefill 372.44 ·
- Ollama · IQ3_XS Decode 80.61 · Prefill 400.97 ·
- llama.cpp · Q4_K_S Decode 108.27 · Prefill 466.84 ·
- vLLM · IQ3_S Decode 128.33 · Prefill 495.87 ·
- vLLM · Q2_K Decode 134.24 · Prefill 721.19 ·
- vLLM · Q2_K Decode 145.89 · Prefill 573.9 ·
- Strata · IQ3_S Decode 180.15 · Prefill 920.56 ·
- Ollama · Q4_K_S Decode 228.86 · Prefill 764.69 ·
- Strata · Q4_K_M Decode 361.74 · Prefill 1475.89 ·
- vLLM · IQ1_S Decode 131.31 · Prefill 466.87 ·
- vLLM · IQ3_S Decode 95.38 · Prefill 330.41 ·
- Strata · IQ3_S Decode 132.6 · Prefill 715.44 ·
- Strata · Q3_K_S Decode 129.13 · Prefill 489.23 ·
- llama.cpp · Q6_K Decode 22.1 · Prefill 165.8 ·
- llama.cpp · Q5_K_S Decode 28.8 · Prefill 216 ·
- llama.cpp · IQ3_S Decode 47.9 · Prefill 359.3 ·
- llama.cpp · IQ4_XS Decode 36 · Prefill 270 ·
- SGLang · Q4_K_M Decode 3.8 · Prefill 28.5 ·
- SGLang · IQ3_S Decode 5.7 · Prefill 42.8 ·
- SGLang · IQ4_XS Decode 4.3 · Prefill 32.3 ·
- vLLM · Q4_K_M Decode 3.9 · Prefill 29.3 ·
- vLLM · IQ3_S Decode 5.9 · Prefill 44.3 ·
- vLLM · IQ4_XS Decode 4.4 · Prefill 33 ·
- vLLM · Q4_K_M Decode 4.4 · Prefill 33 ·
- vLLM · IQ3_S Decode 6.6 · Prefill 49.5 ·
- vLLM · IQ4_XS Decode 5 · Prefill 37.5 ·
- llama.cpp · Q4_K_M Decode 4.1 · Prefill 30.7 ·
- llama.cpp · IQ3_S Decode 6.1 · Prefill 45.8 ·
- llama.cpp · IQ4_XS Decode 4.6 · Prefill 34.5 ·
- Strata · Q4_K_M Decode 8.2 · Prefill 61.5 ·
- Strata · IQ3_XXS Decode 14.2 · Prefill 106.5 ·
- Strata · IQ3_S Decode 12.3 · Prefill 92.3 ·
- Strata · IQ4_XS Decode 9.2 · Prefill 69 ·
- llama.cpp · Q4_K_M Decode 6.4 · Prefill 48 ·
- llama.cpp · IQ3_XXS Decode 11.1 · Prefill 83.3 ·
- llama.cpp · IQ4_XS Decode 7.2 · Prefill 54 ·
- Strata · Q5_K_S Decode 49.2 · Prefill 369 ·
- Strata · IQ3_S Decode 82.2 · Prefill 616.5 ·
Fit check
Physics-based estimate from VRAM and memory bandwidth: pick a model to see which quantizations fit and the expected speed
Formula: tok/s = (bandwidth ÷ model size) × 0.55, adapted from llmfit; see run records for measured numbers
Key specs
- Vendor
- NVIDIA
- Architecture
- Blackwell
- VRAM
- 16 GB
- Memory bandwidth
- 896 GB/s
- TDP
- 300 W
- Release date
- 2025-02-20
Run records
50 configurations
| Model | Framework | Quantization | Decode | Gen time | Samples |
|---|---|---|---|---|---|
| Qwen3.5-4B | llama.cpp | IQ1_S | 182.7 tok/s | — | 1 |
| Qwen3.5-4B | vLLM | IQ4_XS | 131.3 tok/s | — | 1 |
| Qwen3.5-4B | vLLM | Q4_K_M | 116.1 tok/s | — | 1 |
| Qwen3.5-0.8B | vLLM | FP16 | 122 tok/s | — | 1 |
| Qwen3.5-0.8B | vLLM | Q6_K | 174.9 tok/s | — | 1 |
| Qwen3.5-0.8B | Strata | Q5_K_M | 251.2 tok/s | — | 1 |
| Qwen3.5-0.8B | Strata | Q6_K | 237.2 tok/s | — | 1 |
| Qwen3.8-9B | vLLM | IQ3_S | 103.6 tok/s | — | 1 |
| Qwen3.8-9B | vLLM | Q3_K_M | 98.5 tok/s | — | 1 |
| Qwen3.8-9B | Ollama | IQ3_XS | 89.4 tok/s | — | 1 |
| Qwen3.8-9B | Ollama | Q2_K | 98.9 tok/s | — | 1 |
| Qwen3-8B | llama.cpp | Q3_K_M | 110.5 tok/s | — | 1 |
| Qwen3-8B | Ollama | Q4_K_S | 82.5 tok/s | — | 1 |
| gemma-4-12B-it | Ollama | IQ3_XS | 80.6 tok/s | — | 1 |
| Llama-3.1-8B-Instruct | llama.cpp | Q4_K_S | 108.3 tok/s | — | 1 |
| Llama-3.1-8B-Instruct | vLLM | IQ3_S | 128.3 tok/s | — | 1 |
| Llama-3.1-8B-Instruct | vLLM | Q2_K | 134.2 tok/s | — | 1 |
| Llama-3-8B-Instruct | vLLM | Q2_K | 145.9 tok/s | — | 1 |
| Llama-3-8B-Instruct | Strata | IQ3_S | 180.2 tok/s | — | 1 |
| Llama-3.2-3B-Instruct | Ollama | Q4_K_S | 228.9 tok/s | — | 1 |
| Llama-3.2-3B-Instruct | Strata | Q4_K_M | 361.7 tok/s | — | 1 |
| Kimi-K3 | vLLM | IQ1_S | 131.3 tok/s | — | 1 |
| Kimi-K3 | vLLM | IQ3_S | 95.4 tok/s | — | 1 |
| Kimi-K3 | Strata | IQ3_S | 132.6 tok/s | — | 1 |
| Kimi-K3 | Strata | Q3_K_S | 129.1 tok/s | — | 1 |
| Qwen3.5-35B-A3B | llama.cpp | Q6_K | 22.1 tok/s | — | 1 |
| Qwen3.5-35B-A3B | llama.cpp | Q5_K_S | 28.8 tok/s | — | 1 |
| Qwen3.5-35B-A3B | llama.cpp | IQ3_S | 47.9 tok/s | — | 1 |
| Qwen3.5-35B-A3B | llama.cpp | IQ4_XS | 36 tok/s | — | 1 |
| GLM-5.3-Flash | SGLang | Q4_K_M | 3.8 tok/s | — | 1 |
| GLM-5.3-Flash | SGLang | IQ3_S | 5.7 tok/s | — | 1 |
| GLM-5.3-Flash | SGLang | IQ4_XS | 4.3 tok/s | — | 1 |
| GLM-5.3-Flash | vLLM | Q4_K_M | 3.9 tok/s | — | 1 |
| GLM-5.3-Flash | vLLM | IQ3_S | 5.9 tok/s | — | 1 |
| GLM-5.3-Flash | vLLM | IQ4_XS | 4.4 tok/s | — | 1 |
| DeepSeek-V4-Flash | vLLM | Q4_K_M | 4.4 tok/s | — | 1 |
| DeepSeek-V4-Flash | vLLM | IQ3_S | 6.6 tok/s | — | 1 |
| DeepSeek-V4-Flash | vLLM | IQ4_XS | 5 tok/s | — | 1 |
| DeepSeek-V4-Flash | llama.cpp | Q4_K_M | 4.1 tok/s | — | 1 |
| DeepSeek-V4-Flash | llama.cpp | IQ3_S | 6.1 tok/s | — | 1 |
| DeepSeek-V4-Flash | llama.cpp | IQ4_XS | 4.6 tok/s | — | 1 |
| Qwen3.8-Flash-Next | Strata | Q4_K_M | 8.2 tok/s | — | 1 |
| Qwen3.8-Flash-Next | Strata | IQ3_XXS | 14.2 tok/s | — | 1 |
| Qwen3.8-Flash-Next | Strata | IQ3_S | 12.3 tok/s | — | 1 |
| Qwen3.8-Flash-Next | Strata | IQ4_XS | 9.2 tok/s | — | 1 |
| Qwen3.8-Flash-Next | llama.cpp | Q4_K_M | 6.4 tok/s | — | 1 |
| Qwen3.8-Flash-Next | llama.cpp | IQ3_XXS | 11.1 tok/s | — | 1 |
| Qwen3.8-Flash-Next | llama.cpp | IQ4_XS | 7.2 tok/s | — | 1 |
| Qwen3.8-27B | Strata | Q5_K_S | 49.2 tok/s | — | 1 |
| Qwen3.8-27B | Strata | IQ3_S | 82.2 tok/s | — | 1 |