NVIDIA RTX 4070

GPU

Ada Lovelace; 12GB VRAM, 504 GB/s bandwidth, 200W TDP, CUDA cores 5,888. Release date: 2023-04-13. Source: LCZ hardware catalog, with missing fields supplemented from NVIDIA official materials.

15

Run records

8

Models covered

4

Frameworks covered

L2 Cross-framework verified × 15

Evidence mix

Typical measured performance

Published benchmark averages per model × framework × quantization on this hardware.

  1. Strata · IQ3_XS Decode 180.02 · Prefill 970.61 ·
  2. llama.cpp · IQ3_S Decode 227.1 · Prefill 1033.86 ·
  3. vLLM · Q3_K_S Decode 110.14 · Prefill 421.19 ·
  4. vLLM · Q4_K_M Decode 88.36 · Prefill 459.98 ·
  5. vLLM · Q2_K Decode 180.64 · Prefill 850.28 ·
  6. Ollama · Q5_K_S Decode 75.85 · Prefill 366.79 ·
  7. Strata · IQ2_XS Decode 167.79 · Prefill 693.81 ·
  8. Strata · IQ4_XS Decode 136.2 · Prefill 500.55 ·
  9. llama.cpp · IQ1_S Decode 330.62 · Prefill 1805.64 ·
  10. llama.cpp · Q5_K_S Decode 179.43 · Prefill 799.81 ·
  11. vLLM · IQ2_M Decode 318.42 · Prefill 1565.27 ·
  12. vLLM · Q3_K_M Decode 277.32 · Prefill 1224.46 ·
  13. Ollama · Q4_K_M Decode 96.06 · Prefill 324.01 ·
  14. Ollama · Q5_K_M Decode 86.83 · Prefill 374.83 ·
  15. Ollama · IQ2_XS Decode 261.19 · Prefill 1075.45 ·

Fit check

Physics-based estimate from VRAM and memory bandwidth: pick a model to see which quantizations fit and the expected speed

Formula: tok/s = (bandwidth ÷ model size) × 0.55, adapted from llmfit; see run records for measured numbers

Key specs

Vendor
NVIDIA
Architecture
Ada Lovelace
VRAM
12 GB
Memory bandwidth
504 GB/s
TDP
200 W
Release date
2023-04-13

Run records

15 configurations

ModelFrameworkQuantizationDecodeGen timeSamples
Qwen3.5-4BStrataIQ3_XS180 tok/s—1
Qwen3.5-0.8Bllama.cppIQ3_S227.1 tok/s—1
Qwen3-4BvLLMQ3_K_S110.1 tok/s—1
Qwen3-4BvLLMQ4_K_M88.4 tok/s—1
Qwen3-1.7BvLLMQ2_K180.6 tok/s—1
Qwen3-4B-Thinking-2507OllamaQ5_K_S75.8 tok/s—1
Qwen3-4B-Thinking-2507StrataIQ2_XS167.8 tok/s—1
Qwen3-4B-Thinking-2507StrataIQ4_XS136.2 tok/s—1
Qwen2.5-0.5B-Instructllama.cppIQ1_S330.6 tok/s—1
Qwen2.5-0.5B-Instructllama.cppQ5_K_S179.4 tok/s—1
Qwen2.5-0.5B-InstructvLLMIQ2_M318.4 tok/s—1
Qwen2.5-0.5B-InstructvLLMQ3_K_M277.3 tok/s—1
Qwen2.5-Omni-3BOllamaQ4_K_M96.1 tok/s—1
Qwen2.5-Omni-3BOllamaQ5_K_M86.8 tok/s—1
Qwen3-Embedding-0.6BOllamaIQ2_XS261.2 tok/s—1