NVIDIA RTX 4070
GPUAda Lovelace; 12GB VRAM, 504 GB/s bandwidth, 200W TDP, CUDA cores 5,888. Release date: 2023-04-13. Source: LCZ hardware catalog, with missing fields supplemented from NVIDIA official materials.
15
Run records
8
Models covered
4
Frameworks covered
L2 Cross-framework verified × 15
Evidence mix
Typical measured performance
Published benchmark averages per model × framework × quantization on this hardware.
- Strata · IQ3_XS Decode 180.02 · Prefill 970.61 ·
- llama.cpp · IQ3_S Decode 227.1 · Prefill 1033.86 ·
- vLLM · Q3_K_S Decode 110.14 · Prefill 421.19 ·
- vLLM · Q4_K_M Decode 88.36 · Prefill 459.98 ·
- vLLM · Q2_K Decode 180.64 · Prefill 850.28 ·
- Ollama · Q5_K_S Decode 75.85 · Prefill 366.79 ·
- Strata · IQ2_XS Decode 167.79 · Prefill 693.81 ·
- Strata · IQ4_XS Decode 136.2 · Prefill 500.55 ·
- llama.cpp · IQ1_S Decode 330.62 · Prefill 1805.64 ·
- llama.cpp · Q5_K_S Decode 179.43 · Prefill 799.81 ·
- vLLM · IQ2_M Decode 318.42 · Prefill 1565.27 ·
- vLLM · Q3_K_M Decode 277.32 · Prefill 1224.46 ·
- Ollama · Q4_K_M Decode 96.06 · Prefill 324.01 ·
- Ollama · Q5_K_M Decode 86.83 · Prefill 374.83 ·
- Ollama · IQ2_XS Decode 261.19 · Prefill 1075.45 ·
Fit check
Physics-based estimate from VRAM and memory bandwidth: pick a model to see which quantizations fit and the expected speed
Formula: tok/s = (bandwidth ÷ model size) × 0.55, adapted from llmfit; see run records for measured numbers
Key specs
- Vendor
- NVIDIA
- Architecture
- Ada Lovelace
- VRAM
- 12 GB
- Memory bandwidth
- 504 GB/s
- TDP
- 200 W
- Release date
- 2023-04-13
Run records
15 configurations
| Model | Framework | Quantization | Decode | Gen time | Samples |
|---|---|---|---|---|---|
| Qwen3.5-4B | Strata | IQ3_XS | 180 tok/s | — | 1 |
| Qwen3.5-0.8B | llama.cpp | IQ3_S | 227.1 tok/s | — | 1 |
| Qwen3-4B | vLLM | Q3_K_S | 110.1 tok/s | — | 1 |
| Qwen3-4B | vLLM | Q4_K_M | 88.4 tok/s | — | 1 |
| Qwen3-1.7B | vLLM | Q2_K | 180.6 tok/s | — | 1 |
| Qwen3-4B-Thinking-2507 | Ollama | Q5_K_S | 75.8 tok/s | — | 1 |
| Qwen3-4B-Thinking-2507 | Strata | IQ2_XS | 167.8 tok/s | — | 1 |
| Qwen3-4B-Thinking-2507 | Strata | IQ4_XS | 136.2 tok/s | — | 1 |
| Qwen2.5-0.5B-Instruct | llama.cpp | IQ1_S | 330.6 tok/s | — | 1 |
| Qwen2.5-0.5B-Instruct | llama.cpp | Q5_K_S | 179.4 tok/s | — | 1 |
| Qwen2.5-0.5B-Instruct | vLLM | IQ2_M | 318.4 tok/s | — | 1 |
| Qwen2.5-0.5B-Instruct | vLLM | Q3_K_M | 277.3 tok/s | — | 1 |
| Qwen2.5-Omni-3B | Ollama | Q4_K_M | 96.1 tok/s | — | 1 |
| Qwen2.5-Omni-3B | Ollama | Q5_K_M | 86.8 tok/s | — | 1 |
| Qwen3-Embedding-0.6B | Ollama | IQ2_XS | 261.2 tok/s | — | 1 |