Arao AI

NVIDIA RTX 4070 Ti

GPU

Ada Lovelace; 12GB VRAM, 504 GB/s bandwidth, 285W TDP, CUDA cores 7,680. Release date: 2023-01-05. Source: LCZ hardware catalog, with missing fields supplemented from NVIDIA official materials.

34

Run records

15

Models covered

4

Frameworks covered

L2 Cross-framework verified × 34

Evidence mix

Typical measured performance

Published benchmark averages per model × framework × quantization on this hardware.

  1. vLLM · Q4_K_S Decode 161.78 · Prefill 577.24 ·
  2. Ollama · Q6_K Decode 99.87 · Prefill 394.75 ·
  3. llama.cpp · IQ4_XS Decode 272.81 · Prefill 1057.01 ·
  4. vLLM · IQ3_S Decode 341.56 · Prefill 1371.63 ·
  5. vLLM · Q8_0 Decode 188.8 · Prefill 665.28 ·
  6. vLLM · IQ1_S Decode 170.52 · Prefill 868.4 ·
  7. vLLM · Q3_K_S Decode 137.13 · Prefill 568.71 ·
  8. Strata · IQ3_XS Decode 173.97 · Prefill 939.34 ·
  9. vLLM · IQ1_S Decode 188.65 · Prefill 681.35 ·
  10. Strata · IQ2_XS Decode 234.91 · Prefill 843.49 ·
  11. llama.cpp · FP16 Decode 157.49 · Prefill 748.12 ·
  12. Strata · IQ2_M Decode 347.68 · Prefill 1284.54 ·
  13. Strata · Q4_K_M Decode 284.07 · Prefill 1145.96 ·
  14. llama.cpp · Q4_K_M Decode 95.6 · Prefill 473.23 ·
  15. Ollama · Q3_K_S Decode 101.17 · Prefill 440.52 ·
  16. llama.cpp · Q3_K_L Decode 123.86 · Prefill 681.14 ·
  17. Ollama · Q8_0 Decode 68.74 · Prefill 373.18 ·
  18. vLLM · IQ2_XS Decode 105.73 · Prefill 467.89 ·
  19. llama.cpp · IQ3_XS Decode 242.17 · Prefill 959.61 ·
  20. llama.cpp · Q3_K_S Decode 230.59 · Prefill 747.57 ·
  21. vLLM · Q2_K Decode 102.51 · Prefill 448.1 ·
  22. llama.cpp · Q3_K_L Decode 107.42 · Prefill 453.38 ·
  23. llama.cpp · Q4_K_M Decode 104 · Prefill 443.35 ·
  24. vLLM · Q8_0 Decode 66.34 · Prefill 241.62 ·
  25. Strata · Q3_K_L Decode 220.94 · Prefill 762.68 ·
  26. vLLM · Q4_K_M Decode 99.74 · Prefill 485.1 ·
  27. vLLM · Q8_0 Decode 64.87 · Prefill 300.97 ·
  28. vLLM · IQ3_S Decode 273.48 · Prefill 1262.92 ·
  29. vLLM · IQ4_XS Decode 254.8 · Prefill 1153.22 ·
  30. Ollama · Q3_K_S Decode 265.74 · Prefill 1442.65 ·
  31. Ollama · Q8_0 Decode 145.11 · Prefill 753.08 ·
  32. Strata · Q3_K_M Decode 178.43 · Prefill 820.73 ·
  33. Ollama · Q4_K_M Decode 201.81 · Prefill 715.03 ·
  34. Ollama · Q4_K_S Decode 207.17 · Prefill 792.25 ·

Key specs

Vendor
NVIDIA
Architecture
Ada Lovelace
VRAM
12 GB
Memory bandwidth
504 GB/s
TDP
285 W
Release date
2023-01-05

Run records

34 configurations

ModelFrameworkQuantizationDecodeGen timeSamples
Qwen3.5-4BvLLMQ4_K_S161.8 tok/s—1
Qwen3.5-4BOllamaQ6_K99.9 tok/s—1
Qwen3.5-0.8Bllama.cppIQ4_XS272.8 tok/s—1
Qwen3.5-0.8BvLLMIQ3_S341.6 tok/s—1
Qwen3.5-0.8BvLLMQ8_0188.8 tok/s—1
Qwen3.8-9BvLLMIQ1_S170.5 tok/s—1
Qwen3-8BvLLMQ3_K_S137.1 tok/s—1
Qwen3-8BStrataIQ3_XS174 tok/s—1
Llama-3.1-8B-InstructvLLMIQ1_S188.7 tok/s—1
Llama-3-8B-InstructStrataIQ2_XS234.9 tok/s—1
Llama-3.2-3B-Instructllama.cppFP16157.5 tok/s—1
Llama-3.2-3B-InstructStrataIQ2_M347.7 tok/s—1
Llama-3.2-3B-InstructStrataQ4_K_M284.1 tok/s—1
Kimi-K3llama.cppQ4_K_M95.6 tok/s—1
Kimi-K3OllamaQ3_K_S101.2 tok/s—1
Qwen3.5-4Bllama.cppQ3_K_L123.9 tok/s—1
Qwen3.5-4BOllamaQ8_068.7 tok/s—1
Qwen3.5-9BvLLMIQ2_XS105.7 tok/s—1
Qwen3.5-0.8Bllama.cppIQ3_XS242.2 tok/s—1
Qwen3.5-0.8Bllama.cppQ3_K_S230.6 tok/s—1
Qwen3.8-9BvLLMQ2_K102.5 tok/s—1
Qwen3-4Bllama.cppQ3_K_L107.4 tok/s—1
Qwen3-4Bllama.cppQ4_K_M104 tok/s—1
Qwen3-4BvLLMQ8_066.3 tok/s—1
Qwen3-1.7BStrataQ3_K_L220.9 tok/s—1
Qwen3-4B-Thinking-2507vLLMQ4_K_M99.7 tok/s—1
Qwen3-4B-Thinking-2507vLLMQ8_064.9 tok/s—1
Qwen2.5-0.5B-InstructvLLMIQ3_S273.5 tok/s—1
Qwen2.5-0.5B-InstructvLLMIQ4_XS254.8 tok/s—1
Qwen2.5-0.5B-InstructOllamaQ3_K_S265.7 tok/s—1
Qwen2.5-0.5B-InstructOllamaQ8_0145.1 tok/s—1
Qwen2.5-Omni-3BStrataQ3_K_M178.4 tok/s—1
Qwen3-Embedding-0.6BOllamaQ4_K_M201.8 tok/s—1
Qwen3-Embedding-0.6BOllamaQ4_K_S207.2 tok/s—1