Arao AI

NVIDIA RTX 4090

GPU

NVIDIA Ada Lovelace flagship (AD102): 24 GB GDDR6X on a 384-bit bus, 1008 GB/s of memory bandwidth, 450 W board power.

226

Run records

39

Models covered

8

Frameworks covered

L0 Self-reported × 121 · L2 Cross-framework verified × 105

Evidence mix

Typical measured performance

Published benchmark averages per model × framework × quantization on this hardware.

  1. Ollama · Q3_K_M Decode 158.5 · Prefill 719.84 ·
  2. Strata · Q3_K_M Decode 243.76 · Prefill 1261.12 ·
  3. Ollama · Q3_K_S Decode 142.08 · Prefill 478.74 ·
  4. Strata · IQ2_M Decode 230.86 · Prefill 912.94 ·
  5. llama.cpp · Q4_K_M Decode 160.59 · Prefill 829.66 ·
  6. Strata · Q4_K_M Decode 225.92 · Prefill 741.96 ·
  7. Ollama · IQ1_S Decode 130.35 · Prefill 593.51 ·
  8. llama.cpp · IQ1_S Decode 180.81 · Prefill 654.72 ·
  9. llama.cpp · IQ3_XS Decode 136.35 · Prefill 504.8 ·
  10. vLLM · IQ2_XS Decode 179.71 · Prefill 772.31 ·
  11. llama.cpp · Q3_K_S Decode 181.19 · Prefill 930.71 ·
  12. Strata · Q3_K_L Decode 236.63 · Prefill 1225.73 ·
  13. vLLM · Q4_K_S Decode 348.73 · Prefill 1496.87 ·
  14. vLLM · Q8_0 Decode 196.67 · Prefill 827.33 ·
  15. Strata · Q3_K_M Decode 312 · Prefill 1593.74 ·
  16. vLLM · Q3_K_S Decode 144.28 · Prefill 508.23 ·
  17. Strata · Q4_K_M Decode 168.91 · Prefill 879.31 ·
  18. llama.cpp · Q4_K_M Decode 142.71 · Prefill 813.73 ·
  19. vLLM · IQ3_S Decode 161.14 · Prefill 813.02 ·
  20. llama.cpp · Q3_K_L Decode 114.65 · Prefill 473.3 ·
  21. Ollama · Q4_K_M Decode 87.43 · Prefill 399.69 ·
  22. Ollama · Q6_K Decode 70.15 · Prefill 334.66 ·
  23. vLLM · IQ2_XS Decode 205.12 · Prefill 1126.08 ·
  24. Strata · Q4_K_M Decode 191.74 · Prefill 642.2 ·
  25. Strata · Q6_K Decode 151.86 · Prefill 574.29 ·
  26. llama.cpp · Q5_K_S Decode 118.71 · Prefill 394.1 ·
  27. vLLM · IQ3_S Decode 177.41 · Prefill 791.84 ·
  28. vLLM · Q4_K_S Decode 157.4 · Prefill 646.64 ·
  29. vLLM · IQ3_S Decode 327.57 · Prefill 1515.42 ·
  30. Ollama · Q4_K_S Decode 336.75 · Prefill 1261.4 ·
  31. Ollama · Q5_K_M Decode 263.12 · Prefill 1197.54 ·
  32. llama.cpp · Q3_K_M Decode 122.06 · Prefill 409.59 ·
  33. llama.cpp · FP16 Decode 79.5 · Prefill 355.19 ·
  34. llama.cpp · IQ4_XS Decode 170.35 · Prefill 636.3 ·
  35. vLLM · IQ2_M Decode 248.85 · Prefill 1225.63 ·
  36. vLLM · Q6_K Decode 133.61 · Prefill 629.89 ·
  37. llama.cpp · IQ1_S Decode 373.87 · Prefill 1737.06 ·
  38. llama.cpp · Q3_K_S Decode 364.6 · Prefill 1679.71 ·
  39. vLLM · FP16 Decode 161.03 · Prefill 685.23 ·
  40. vLLM · IQ1_S Decode 328.9 · Prefill 1532.17 ·
  41. vLLM · IQ4_XS Decode 131.27 · Prefill 643.79 ·
  42. llama.cpp · IQ1_S Decode 187.88 · Prefill 983.15 ·
  43. llama.cpp · Q8_0 Decode 87.18 · Prefill 465.1 ·
  44. vLLM · Q2_K Decode 164.91 · Prefill 671.93 ·
  45. llama.cpp · IQ2_XS Decode 152.32 · Prefill 683.44 ·
  46. Strata · Q5_K_S Decode 117.55 · Prefill 384.33 ·
  47. llama.cpp · Q5_K_M Decode 128.55 · Prefill 827.45 ·
  48. vLLM · Q4_K_M Decode 142.74 · Prefill 510.45 ·
  49. llama.cpp · Q4_K_M Decode 146.97 · Prefill 902.58 ·
  50. llama.cpp · Q4_K_S Decode 134.4 · Prefill 594.59 ·

Fit check

Physics-based estimate from VRAM and memory bandwidth: pick a model to see which quantizations fit and the expected speed

Formula: tok/s = (bandwidth ÷ model size) × 0.55, adapted from llmfit; see run records for measured numbers

Key specs

Vendor
NVIDIA
Architecture
Ada Lovelace
VRAM
24 GB
Memory bandwidth
1008 GB/s
TDP
450 W
Release date
2022-10-12

Run records

50 configurations

ModelFrameworkQuantizationDecodeGen timeSamples
gemma-4-E4B-itOllamaQ3_K_M158.5 tok/s—1
gemma-4-E4B-itStrataQ3_K_M243.8 tok/s—1
Ministral-3-8B-Instruct-2512OllamaQ3_K_S142.1 tok/s—1
Ministral-3-8B-Instruct-2512StrataIQ2_M230.9 tok/s—1
NVIDIA-Nemotron-3.5-Bolt-E4B-4GBllama.cppQ4_K_M160.6 tok/s—1
NVIDIA-Nemotron-3.5-Bolt-E4B-4GBStrataQ4_K_M225.9 tok/s—1
mammoth-1OllamaIQ1_S130.3 tok/s—1
elephant-1llama.cppIQ1_S180.8 tok/s—1
elephant-1llama.cppIQ3_XS136.3 tok/s—1
elephant-1vLLMIQ2_XS179.7 tok/s—1
Qwen3.5-4Bllama.cppQ3_K_S181.2 tok/s—1
Qwen3.5-4BStrataQ3_K_L236.6 tok/s—1
Qwen3.5-0.8BvLLMQ4_K_S348.7 tok/s—1
Qwen3.5-0.8BvLLMQ8_0196.7 tok/s—1
Qwen3.5-0.8BStrataQ3_K_M312 tok/s—1
Qwen3.8-9BvLLMQ3_K_S144.3 tok/s—1
Qwen3.8-9BStrataQ4_K_M168.9 tok/s—1
Qwen3-8Bllama.cppQ4_K_M142.7 tok/s—2
Qwen3-8BvLLMIQ3_S161.1 tok/s—1
gemma-4-12B-itllama.cppQ3_K_L114.7 tok/s—1
gemma-4-12B-itOllamaQ4_K_M87.4 tok/s—1
gemma-4-12B-itOllamaQ6_K70.2 tok/s—2
Llama-3.1-8B-InstructvLLMIQ2_XS205.1 tok/s—1
Llama-3.1-8B-InstructStrataQ4_K_M191.7 tok/s—1
Llama-3.1-8B-InstructStrataQ6_K151.9 tok/s—1
Llama-3-8B-Instructllama.cppQ5_K_S118.7 tok/s—1
Llama-3-8B-InstructvLLMIQ3_S177.4 tok/s—1
Llama-3-8B-InstructvLLMQ4_K_S157.4 tok/s—1
Llama-3.2-3B-InstructvLLMIQ3_S327.6 tok/s—2
Llama-3.2-3B-InstructOllamaQ4_K_S336.8 tok/s—1
Llama-3.2-3B-InstructOllamaQ5_K_M263.1 tok/s—1
DeepSeek-R1-Distill-Qwen-14Bllama.cppQ3_K_M122.1 tok/s—1
Qwen3.5-4Bllama.cppFP1679.5 tok/s—1
Qwen3.5-4Bllama.cppIQ4_XS170.3 tok/s—1
Qwen3.5-4BvLLMIQ2_M248.8 tok/s—1
Qwen3.5-4BvLLMQ6_K133.6 tok/s—1
Qwen3.5-0.8Bllama.cppIQ1_S373.9 tok/s—1
Qwen3.5-0.8Bllama.cppQ3_K_S364.6 tok/s—1
Qwen3.5-0.8BvLLMFP16161 tok/s—1
Qwen3.5-0.8BvLLMIQ1_S328.9 tok/s—1
Qwen3.8-9BvLLMIQ4_XS131.3 tok/s—1
Qwen3-8Bllama.cppIQ1_S187.9 tok/s—1
Qwen3-8Bllama.cppQ8_087.2 tok/s—3
Qwen3-8BvLLMQ2_K164.9 tok/s—1
gemma-4-12B-itllama.cppIQ2_XS152.3 tok/s—1
gemma-4-12B-itStrataQ5_K_S117.5 tok/s—1
Llama-3.1-8B-Instructllama.cppQ5_K_M128.6 tok/s—2
Llama-3.1-8B-InstructvLLMQ4_K_M142.7 tok/s—1
Llama-3-8B-Instructllama.cppQ4_K_M147 tok/s—2
Llama-3-8B-Instructllama.cppQ4_K_S134.4 tok/s—1