Arao AI

NVIDIA RTX 4080

GPU

Ada Lovelace; 16GB VRAM, 717 GB/s bandwidth, 320W TDP, CUDA cores 9,728. Release date: 2022-11-16. Source: LCZ hardware catalog, with missing fields supplemented from NVIDIA official materials.

52

Run records

17

Models covered

4

Frameworks covered

L2 Cross-framework verified × 52

Evidence mix

Typical measured performance

Published benchmark averages per model × framework × quantization on this hardware.

  1. Ollama · IQ2_XS Decode 194.9 · Prefill 903.61 ·
  2. Strata · Q6_K Decode 181.02 · Prefill 665.46 ·
  3. vLLM · Q3_K_S Decode 378.63 · Prefill 1756.55 ·
  4. vLLM · Q5_K_M Decode 304.13 · Prefill 1485.96 ·
  5. Strata · IQ1_S Decode 334.33 · Prefill 1448.23 ·
  6. Strata · Q8_0 Decode 328.32 · Prefill 1297.3 ·
  7. Ollama · IQ2_XS Decode 160.5 · Prefill 666.22 ·
  8. Ollama · IQ4_XS Decode 123.16 · Prefill 512.65 ·
  9. Strata · IQ4_XS Decode 177.56 · Prefill 752.37 ·
  10. Ollama · IQ3_S Decode 137.8 · Prefill 706.29 ·
  11. Ollama · IQ3_XS Decode 158.58 · Prefill 727 ·
  12. Strata · IQ3_S Decode 234.8 · Prefill 1203.48 ·
  13. Strata · IQ4_XS Decode 189.04 · Prefill 930.37 ·
  14. Ollama · Q3_K_S Decode 117.06 · Prefill 479.42 ·
  15. Strata · IQ1_S Decode 247.32 · Prefill 1242.06 ·
  16. llama.cpp · Q3_K_L Decode 160.83 · Prefill 637.28 ·
  17. Strata · IQ4_XS Decode 248.44 · Prefill 973.3 ·
  18. Strata · Q3_K_M Decode 238.79 · Prefill 1287.02 ·
  19. vLLM · IQ3_XS Decode 205.69 · Prefill 981.9 ·
  20. vLLM · Q5_K_S Decode 132.58 · Prefill 716.19 ·
  21. Strata · Q3_K_M Decode 260.87 · Prefill 961.11 ·
  22. llama.cpp · Q3_K_S Decode 351.55 · Prefill 1651.53 ·
  23. llama.cpp · Q5_K_S Decode 338.44 · Prefill 1522.16 ·
  24. Strata · Q4_K_M Decode 341.28 · Prefill 1116.7 ·
  25. Ollama · IQ2_M Decode 163.57 · Prefill 753.22 ·
  26. Ollama · IQ3_XS Decode 139.12 · Prefill 753.5 ·
  27. Ollama · Q5_K_M Decode 91.51 · Prefill 334.92 ·
  28. Strata · FP16 Decode 79.66 · Prefill 266.23 ·
  29. Strata · Q4_K_M Decode 165.81 · Prefill 742.77 ·
  30. vLLM · Q2_K Decode 151.45 · Prefill 529.44 ·
  31. vLLM · Q3_K_M Decode 130.67 · Prefill 528.13 ·
  32. Ollama · Q4_K_M Decode 95.12 · Prefill 401.62 ·
  33. Ollama · Q4_K_S Decode 71.83 · Prefill 361.81 ·
  34. vLLM · Q6_K Decode 169.83 · Prefill 688.66 ·
  35. Strata · Q3_K_S Decode 370.12 · Prefill 1878.16 ·
  36. llama.cpp · IQ1_S Decode 141.07 · Prefill 456.51 ·
  37. llama.cpp · IQ3_S Decode 105.08 · Prefill 472.34 ·
  38. vLLM · IQ1_S Decode 150.38 · Prefill 727.2 ·
  39. Ollama · IQ4_XS Decode 104.26 · Prefill 504.67 ·
  40. vLLM · IQ2_M Decode 224.82 · Prefill 990.41 ·
  41. vLLM · Q3_K_M Decode 197.14 · Prefill 742.62 ·
  42. Ollama · Q4_K_S Decode 163.88 · Prefill 605 ·
  43. Ollama · Q8_0 Decode 101.94 · Prefill 440.86 ·
  44. vLLM · Q5_K_M Decode 92.54 · Prefill 405.16 ·
  45. vLLM · IQ2_XS Decode 372.57 · Prefill 1858.32 ·
  46. Strata · IQ1_S Decode 264.72 · Prefill 1096.35 ·
  47. Strata · IQ2_XS Decode 273.01 · Prefill 1309.97 ·
  48. vLLM · IQ1_S Decode 305.89 · Prefill 1188.81 ·
  49. vLLM · Q3_K_S Decode 310.84 · Prefill 1177.49 ·
  50. Strata · Q3_K_S Decode 388.23 · Prefill 2020.82 ·

Key specs

Vendor
NVIDIA
Architecture
Ada Lovelace
VRAM
16 GB
Memory bandwidth
717 GB/s
TDP
320 W
Release date
2022-11-16

Run records

50 configurations

ModelFrameworkQuantizationDecodeGen timeSamples
Qwen3.5-4BOllamaIQ2_XS194.9 tok/s—2
Qwen3.5-4BStrataQ6_K181 tok/s—1
Qwen3.5-0.8BvLLMQ3_K_S378.6 tok/s—1
Qwen3.5-0.8BvLLMQ5_K_M304.1 tok/s—1
Qwen3.5-0.8BStrataIQ1_S334.3 tok/s—2
Qwen3.5-0.8BStrataQ8_0328.3 tok/s—1
Qwen3.8-9BOllamaIQ2_XS160.5 tok/s—1
Qwen3.8-9BOllamaIQ4_XS123.2 tok/s—1
Qwen3.8-9BStrataIQ4_XS177.6 tok/s—1
Qwen3-8BOllamaIQ3_S137.8 tok/s—1
Qwen3-8BOllamaIQ3_XS158.6 tok/s—1
Qwen3-8BStrataIQ3_S234.8 tok/s—1
Qwen3-8BStrataIQ4_XS189 tok/s—1
gemma-4-12B-itOllamaQ3_K_S117.1 tok/s—1
gemma-4-12B-itStrataIQ1_S247.3 tok/s—1
Llama-3.1-8B-Instructllama.cppQ3_K_L160.8 tok/s—1
Llama-3.1-8B-InstructStrataIQ4_XS248.4 tok/s—1
Llama-3.1-8B-InstructStrataQ3_K_M238.8 tok/s—1
Llama-3-8B-InstructvLLMIQ3_XS205.7 tok/s—1
Llama-3-8B-InstructvLLMQ5_K_S132.6 tok/s—1
Llama-3-8B-InstructStrataQ3_K_M260.9 tok/s—1
Llama-3.2-3B-Instructllama.cppQ3_K_S351.6 tok/s—1
Llama-3.2-3B-Instructllama.cppQ5_K_S338.4 tok/s—1
Llama-3.2-3B-InstructStrataQ4_K_M341.3 tok/s—1
DeepSeek-R1-Distill-Qwen-14BOllamaIQ2_M163.6 tok/s—1
Kimi-K3OllamaIQ3_XS139.1 tok/s—1
Kimi-K3OllamaQ5_K_M91.5 tok/s—1
Kimi-K3StrataFP1679.7 tok/s—1
Kimi-K3StrataQ4_K_M165.8 tok/s—1
Qwen3.5-4BvLLMQ2_K151.4 tok/s—1
Qwen3.5-4BvLLMQ3_K_M130.7 tok/s—1
Qwen3.5-4BOllamaQ4_K_M95.1 tok/s—1
Qwen3.5-9BOllamaQ4_K_S71.8 tok/s—1
Qwen3.5-0.8BvLLMQ6_K169.8 tok/s—1
Qwen3.5-0.8BStrataQ3_K_S370.1 tok/s—1
Qwen3-8Bllama.cppIQ1_S141.1 tok/s—1
Qwen3-8Bllama.cppIQ3_S105.1 tok/s—1
Qwen3-8BvLLMIQ1_S150.4 tok/s—1
Qwen3-4BOllamaIQ4_XS104.3 tok/s—1
Qwen3-1.7BvLLMIQ2_M224.8 tok/s—1
Qwen3-1.7BvLLMQ3_K_M197.1 tok/s—1
Qwen3-1.7BOllamaQ4_K_S163.9 tok/s—1
Qwen3-1.7BOllamaQ8_0101.9 tok/s—1
Qwen3-4B-Thinking-2507vLLMQ5_K_M92.5 tok/s—1
Qwen2.5-0.5B-InstructvLLMIQ2_XS372.6 tok/s—1
Qwen2.5-Omni-3BStrataIQ1_S264.7 tok/s—1
Qwen2.5-Omni-3BStrataIQ2_XS273 tok/s—1
Qwen3-Embedding-0.6BvLLMIQ1_S305.9 tok/s—1
Qwen3-Embedding-0.6BvLLMQ3_K_S310.8 tok/s—1
Qwen3-Embedding-0.6BStrataQ3_K_S388.2 tok/s—1