Arao AI

NVIDIA RTX 4060 Ti 16GB

GPU

NVIDIA Ada Lovelace (AD106): 16 GB GDDR6 on a narrow 128-bit bus, 288 GB/s of memory bandwidth, 165 W board power. Bandwidth is limited, but 16 GB of VRAM makes it more useful than its price peers for slightly larger quantized models.

44

Run records

17

Models covered

4

Frameworks covered

L2 Cross-framework verified × 44

Evidence mix

Typical measured performance

Published benchmark averages per model × framework × quantization on this hardware.

  1. llama.cpp · Q3_K_S Decode 147.52 · Prefill 627.05 ·
  2. Ollama · Q3_K_M Decode 144.59 · Prefill 690.26 ·
  3. vLLM · FP16 Decode 128.16 · Prefill 630.91 ·
  4. vLLM · Q4_K_S Decode 240.41 · Prefill 996.87 ·
  5. Ollama · Q5_K_S Decode 177.61 · Prefill 661.12 ·
  6. Ollama · Q6_K Decode 170.01 · Prefill 704.45 ·
  7. vLLM · IQ4_XS Decode 104.81 · Prefill 479.61 ·
  8. vLLM · Q3_K_M Decode 106.76 · Prefill 483.03 ·
  9. Strata · IQ2_XS Decode 182.07 · Prefill 797.96 ·
  10. llama.cpp · IQ3_XS Decode 120.55 · Prefill 447.58 ·
  11. llama.cpp · Q2_K Decode 129.58 · Prefill 532.02 ·
  12. vLLM · Q3_K_L Decode 108.14 · Prefill 543.11 ·
  13. vLLM · Q3_K_S Decode 104.24 · Prefill 463.62 ·
  14. Ollama · Q3_K_L Decode 118.42 · Prefill 384.95 ·
  15. Strata · IQ4_XS Decode 154.65 · Prefill 522.45 ·
  16. Strata · IQ3_S Decode 187.1 · Prefill 768.08 ·
  17. llama.cpp · IQ3_S Decode 326.89 · Prefill 1104.91 ·
  18. vLLM · Q2_K Decode 350.23 · Prefill 1317.38 ·
  19. vLLM · IQ2_XS Decode 125.25 · Prefill 439.59 ·
  20. vLLM · Q5_K_S Decode 82.77 · Prefill 438.83 ·
  21. Ollama · IQ3_XS Decode 90.69 · Prefill 447.55 ·
  22. Ollama · IQ4_XS Decode 88.03 · Prefill 388.48 ·
  23. Strata · Q2_K Decode 195.73 · Prefill 812.84 ·
  24. Ollama · IQ3_S Decode 73.08 · Prefill 389.31 ·
  25. vLLM · IQ4_XS Decode 206.77 · Prefill 955.7 ·
  26. vLLM · Q3_K_S Decode 239.77 · Prefill 1073.87 ·
  27. Ollama · IQ1_S Decode 259.18 · Prefill 1280.06 ·
  28. vLLM · IQ2_M Decode 110.8 · Prefill 556.06 ·
  29. Ollama · Q4_K_M Decode 65.8 · Prefill 285.42 ·
  30. vLLM · IQ3_XS Decode 103.03 · Prefill 480.63 ·
  31. Strata · Q6_K Decode 108.91 · Prefill 545.63 ·
  32. Ollama · IQ1_S Decode 203.18 · Prefill 859.41 ·
  33. Ollama · IQ2_XS Decode 184.49 · Prefill 897.89 ·
  34. vLLM · IQ3_S Decode 108.19 · Prefill 350.2 ·
  35. vLLM · Q3_K_M Decode 119.54 · Prefill 394.34 ·
  36. Strata · IQ3_XS Decode 160.63 · Prefill 538.1 ·
  37. Strata · Q3_K_L Decode 140.78 · Prefill 483.92 ·
  38. llama.cpp · IQ1_S Decode 337.12 · Prefill 1106.41 ·
  39. llama.cpp · Q2_K Decode 164.89 · Prefill 695.02 ·
  40. llama.cpp · Q4_K_S Decode 221.58 · Prefill 758 ·
  41. vLLM · IQ4_XS Decode 223.15 · Prefill 875.24 ·
  42. llama.cpp · IQ2_M Decode 81.49 · Prefill 352.61 ·
  43. Ollama · IQ1_S Decode 78.54 · Prefill 337.36 ·
  44. Ollama · Q2_K Decode 74.86 · Prefill 405.27 ·

Fit check

Physics-based estimate from VRAM and memory bandwidth: pick a model to see which quantizations fit and the expected speed

Formula: tok/s = (bandwidth ÷ model size) × 0.55, adapted from llmfit; see run records for measured numbers

Key specs

Vendor
NVIDIA
Architecture
Ada Lovelace
VRAM
16 GB
Memory bandwidth
288 GB/s
TDP
165 W
Release date
2023-07-18

Run records

44 configurations

ModelFrameworkQuantizationDecodeGen timeSamples
Qwen3.5-4Bllama.cppQ3_K_S147.5 tok/s—1
Qwen3.5-4BOllamaQ3_K_M144.6 tok/s—1
Qwen3.5-0.8BvLLMFP16128.2 tok/s—1
Qwen3.5-0.8BvLLMQ4_K_S240.4 tok/s—1
Qwen3.5-0.8BOllamaQ5_K_S177.6 tok/s—1
Qwen3.5-0.8BOllamaQ6_K170 tok/s—1
Qwen3.8-9BvLLMIQ4_XS104.8 tok/s—1
Qwen3.8-9BvLLMQ3_K_M106.8 tok/s—1
Qwen3.8-9BStrataIQ2_XS182.1 tok/s—1
Qwen3-8Bllama.cppIQ3_XS120.5 tok/s—1
Qwen3-8Bllama.cppQ2_K129.6 tok/s—1
Qwen3-8BvLLMQ3_K_L108.1 tok/s—1
gemma-4-12B-itvLLMQ3_K_S104.2 tok/s—1
Llama-3.1-8B-InstructOllamaQ3_K_L118.4 tok/s—1
Llama-3.1-8B-InstructStrataIQ4_XS154.7 tok/s—1
Llama-3-8B-InstructStrataIQ3_S187.1 tok/s—1
Llama-3.2-3B-Instructllama.cppIQ3_S326.9 tok/s—1
Llama-3.2-3B-InstructvLLMQ2_K350.2 tok/s—1
DeepSeek-R1-Distill-Qwen-14BvLLMIQ2_XS125.3 tok/s—1
Kimi-K3vLLMQ5_K_S82.8 tok/s—1
Kimi-K3OllamaIQ3_XS90.7 tok/s—1
Kimi-K3OllamaIQ4_XS88 tok/s—1
Qwen3.5-4BStrataQ2_K195.7 tok/s—1
Qwen3.5-9BOllamaIQ3_S73.1 tok/s—1
Qwen3.5-0.8BvLLMIQ4_XS206.8 tok/s—1
Qwen3.5-0.8BvLLMQ3_K_S239.8 tok/s—1
Qwen3.5-0.8BOllamaIQ1_S259.2 tok/s—1
Qwen3.8-9BvLLMIQ2_M110.8 tok/s—1
Qwen3.8-9BOllamaQ4_K_M65.8 tok/s—1
Qwen3-8BvLLMIQ3_XS103 tok/s—1
Qwen3-4BStrataQ6_K108.9 tok/s—1
Qwen3-1.7BOllamaIQ1_S203.2 tok/s—1
Qwen3-1.7BOllamaIQ2_XS184.5 tok/s—1
Qwen3-4B-Thinking-2507vLLMIQ3_S108.2 tok/s—1
Qwen3-4B-Thinking-2507vLLMQ3_K_M119.5 tok/s—1
Qwen3-4B-Thinking-2507StrataIQ3_XS160.6 tok/s—1
Qwen3-4B-Thinking-2507StrataQ3_K_L140.8 tok/s—1
Qwen2.5-0.5B-Instructllama.cppIQ1_S337.1 tok/s—1
Qwen2.5-Omni-3Bllama.cppQ2_K164.9 tok/s—1
Qwen3-Embedding-0.6Bllama.cppQ4_K_S221.6 tok/s—1
Qwen3-Embedding-0.6BvLLMIQ4_XS223.2 tok/s—1
gemma-4-12B-itllama.cppIQ2_M81.5 tok/s—1
gemma-4-12B-itOllamaIQ1_S78.5 tok/s—1
gemma-4-12B-itOllamaQ2_K74.9 tok/s—1