Arao AI

NVIDIA RTX 4080 Super

GPU

NVIDIA Ada Lovelace (AD103): 16 GB GDDR6X on a 256-bit bus, 736 GB/s of memory bandwidth, 320 W board power.

30

Run records

5

Models covered

4

Frameworks covered

L0 Self-reported × 30

Evidence mix

Typical measured performance

Published benchmark averages per model × framework × quantization on this hardware.

  1. llama.cpp · Q6_K Decode 18.2 · Prefill 136.5 ·
  2. llama.cpp · Q5_K_S Decode 23.6 · Prefill 177 ·
  3. llama.cpp · IQ3_S Decode 39.3 · Prefill 294.8 ·
  4. llama.cpp · IQ4_XS Decode 29.6 · Prefill 222 ·
  5. SGLang · Q4_K_M Decode 3.1 · Prefill 23.3 ·
  6. SGLang · IQ3_S Decode 4.7 · Prefill 35.3 ·
  7. SGLang · IQ4_XS Decode 3.5 · Prefill 26.3 ·
  8. vLLM · Q4_K_M Decode 3.2 · Prefill 24 ·
  9. vLLM · IQ3_S Decode 4.8 · Prefill 36 ·
  10. vLLM · IQ4_XS Decode 3.6 · Prefill 27 ·
  11. vLLM · Q4_K_M Decode 3.6 · Prefill 27 ·
  12. vLLM · IQ3_S Decode 5.4 · Prefill 40.5 ·
  13. vLLM · IQ4_XS Decode 4.1 · Prefill 30.7 ·
  14. llama.cpp · Q4_K_M Decode 3.3 · Prefill 24.8 ·
  15. llama.cpp · IQ3_S Decode 5 · Prefill 37.5 ·
  16. llama.cpp · IQ4_XS Decode 3.7 · Prefill 27.8 ·
  17. Strata · Q5_K_S Decode 40.4 · Prefill 303 ·
  18. Strata · IQ3_S Decode 67.5 · Prefill 506.3 ·
  19. Strata · IQ4_XS Decode 50.4 · Prefill 378 ·
  20. llama.cpp · IQ4_XS Decode 39.3 · Prefill 294.8 ·
  21. llama.cpp · Q5_K_S Decode 31.5 · Prefill 236.3 ·
  22. llama.cpp · IQ3_XXS Decode 60.4 · Prefill 453 ·
  23. llama.cpp · IQ3_S Decode 52.7 · Prefill 395.3 ·
  24. llama.cpp · Q6_K Decode 24.2 · Prefill 181.5 ·
  25. vLLM · IQ2_XS Decode 59.6 · Prefill 447 ·
  26. vLLM · Q5_K_M Decode 27.2 · Prefill 204 ·
  27. vLLM · Q4_K_M Decode 33.3 · Prefill 249.7 ·
  28. llama.cpp · IQ2_XS Decode 54.7 · Prefill 410.3 ·
  29. llama.cpp · Q5_K_M Decode 25 · Prefill 187.5 ·
  30. llama.cpp · Q4_K_M Decode 30.5 · Prefill 228.8 ·

Key specs

Vendor
NVIDIA
Architecture
Ada Lovelace
VRAM
16 GB
Memory bandwidth
736 GB/s
TDP
320 W
Release date
2024-01-31

Run records

30 configurations

ModelFrameworkQuantizationDecodeGen timeSamples
Qwen3.5-35B-A3Bllama.cppQ6_K18.2 tok/s—1
Qwen3.5-35B-A3Bllama.cppQ5_K_S23.6 tok/s—1
Qwen3.5-35B-A3Bllama.cppIQ3_S39.3 tok/s—1
Qwen3.5-35B-A3Bllama.cppIQ4_XS29.6 tok/s—1
GLM-5.3-FlashSGLangQ4_K_M3.1 tok/s—1
GLM-5.3-FlashSGLangIQ3_S4.7 tok/s—1
GLM-5.3-FlashSGLangIQ4_XS3.5 tok/s—1
GLM-5.3-FlashvLLMQ4_K_M3.2 tok/s—1
GLM-5.3-FlashvLLMIQ3_S4.8 tok/s—1
GLM-5.3-FlashvLLMIQ4_XS3.6 tok/s—1
DeepSeek-V4-FlashvLLMQ4_K_M3.6 tok/s—1
DeepSeek-V4-FlashvLLMIQ3_S5.4 tok/s—1
DeepSeek-V4-FlashvLLMIQ4_XS4.1 tok/s—1
DeepSeek-V4-Flashllama.cppQ4_K_M3.3 tok/s—1
DeepSeek-V4-Flashllama.cppIQ3_S5 tok/s—1
DeepSeek-V4-Flashllama.cppIQ4_XS3.7 tok/s—1
Qwen3.8-27BStrataQ5_K_S40.4 tok/s—1
Qwen3.8-27BStrataIQ3_S67.5 tok/s—1
Qwen3.8-27BStrataIQ4_XS50.4 tok/s—1
Qwen3.8-27Bllama.cppIQ4_XS39.3 tok/s—1
Qwen3.8-27Bllama.cppQ5_K_S31.5 tok/s—1
Qwen3.8-27Bllama.cppIQ3_XXS60.4 tok/s—1
Qwen3.8-27Bllama.cppIQ3_S52.7 tok/s—1
Qwen3.8-27Bllama.cppQ6_K24.2 tok/s—1
gemma-4-31B-itvLLMIQ2_XS59.6 tok/s—1
gemma-4-31B-itvLLMQ5_K_M27.2 tok/s—1
gemma-4-31B-itvLLMQ4_K_M33.3 tok/s—1
gemma-4-31B-itllama.cppIQ2_XS54.7 tok/s—1
gemma-4-31B-itllama.cppQ5_K_M25 tok/s—1
gemma-4-31B-itllama.cppQ4_K_M30.5 tok/s—1