AMD Radeon AI PRO R9700

GPU

33

Run records

13

Models covered

4

Frameworks covered

L2 Cross-framework verified × 33

Evidence mix

Typical measured performance

Published benchmark averages per model × framework × quantization on this hardware.

  1. llama.cpp · Q3_K_M Decode 183.89 · Prefill 727.67 ·
  2. vLLM · IQ3_XS Decode 208.55 · Prefill 738.48 ·
  3. vLLM · Q5_K_M Decode 122.86 · Prefill 611.36 ·
  4. vLLM · Q4_K_M Decode 98.86 · Prefill 492.64 ·
  5. vLLM · Q6_K Decode 76.56 · Prefill 349.76 ·
  6. llama.cpp · IQ2_XS Decode 394.48 · Prefill 1743.78 ·
  7. Ollama · IQ1_S Decode 368.92 · Prefill 1396.86 ·
  8. Ollama · Q4_K_S Decode 284.75 · Prefill 1200.05 ·
  9. Ollama · Q3_K_M Decode 123.24 · Prefill 659.52 ·
  10. Ollama · Q3_K_S Decode 110.64 · Prefill 546.12 ·
  11. llama.cpp · IQ2_M Decode 154.04 · Prefill 793.94 ·
  12. llama.cpp · Q3_K_S Decode 137.81 · Prefill 521.21 ·
  13. Strata · IQ1_S Decode 246.4 · Prefill 1052.77 ·
  14. Strata · IQ3_XS Decode 190.7 · Prefill 730.63 ·
  15. Strata · Q4_K_S Decode 204.13 · Prefill 1042.53 ·
  16. vLLM · Q3_K_S Decode 272.77 · Prefill 1363 ·
  17. Ollama · IQ4_XS Decode 192.57 · Prefill 691.62 ·
  18. vLLM · Q4_K_S Decode 158.57 · Prefill 583.28 ·
  19. vLLM · Q5_K_M Decode 111.48 · Prefill 571.17 ·
  20. vLLM · IQ3_XS Decode 346.96 · Prefill 1652.99 ·
  21. vLLM · Q5_K_S Decode 310.96 · Prefill 1151.48 ·
  22. Strata · Q3_K_M Decode 294.81 · Prefill 1272.19 ·
  23. Strata · Q5_K_M Decode 384.75 · Prefill 2096.67 ·
  24. llama.cpp · IQ4_XS Decode 183.61 · Prefill 748.24 ·
  25. llama.cpp · Q2_K Decode 223.75 · Prefill 955.8 ·
  26. Ollama · IQ2_XS Decode 228.57 · Prefill 1151.43 ·
  27. Ollama · IQ4_XS Decode 178.25 · Prefill 834.26 ·
  28. llama.cpp · IQ2_XS Decode 354.83 · Prefill 1505.4 ·
  29. llama.cpp · IQ4_XS Decode 342.35 · Prefill 1387.76 ·
  30. vLLM · IQ2_M Decode 125.41 · Prefill 606.9 ·
  31. vLLM · IQ3_S Decode 96.64 · Prefill 451.11 ·
  32. vLLM · Q5_K_M Decode 71.18 · Prefill 263.48 ·
  33. Strata · IQ3_XS Decode 151.48 · Prefill 693.94 ·

Fit check

Physics-based estimate from VRAM and memory bandwidth: pick a model to see which quantizations fit and the expected speed

Formula: tok/s = (bandwidth ÷ model size) × 0.55, adapted from llmfit; see run records for measured numbers

Key specs

Vendor
AMD
VRAM
32 GB
Memory bandwidth
512 GB/s

Run records

33 configurations

ModelFrameworkQuantizationDecodeGen timeSamples
Qwen3.5-4Bllama.cppQ3_K_M183.9 tok/s—1
Qwen3.5-4BvLLMIQ3_XS208.6 tok/s—1
Qwen3.5-4BvLLMQ5_K_M122.9 tok/s—1
Qwen3.5-9BvLLMQ4_K_M98.9 tok/s—1
Qwen3.5-9BvLLMQ6_K76.6 tok/s—1
Qwen3.5-0.8Bllama.cppIQ2_XS394.5 tok/s—1
Qwen3.5-0.8BOllamaIQ1_S368.9 tok/s—1
Qwen3.5-0.8BOllamaQ4_K_S284.8 tok/s—1
Qwen3.8-9BOllamaQ3_K_M123.2 tok/s—1
Qwen3.8-9BOllamaQ3_K_S110.6 tok/s—1
Qwen3-8Bllama.cppIQ2_M154 tok/s—1
Qwen3-8Bllama.cppQ3_K_S137.8 tok/s—1
Qwen3-8BStrataIQ1_S246.4 tok/s—1
Qwen3-8BStrataIQ3_XS190.7 tok/s—1
Qwen3-4BStrataQ4_K_S204.1 tok/s—1
Qwen3-1.7BvLLMQ3_K_S272.8 tok/s—1
Qwen3-1.7BOllamaIQ4_XS192.6 tok/s—1
Qwen3-4B-Thinking-2507vLLMQ4_K_S158.6 tok/s—1
Qwen3-4B-Thinking-2507vLLMQ5_K_M111.5 tok/s—1
Qwen2.5-0.5B-InstructvLLMIQ3_XS347 tok/s—1
Qwen2.5-0.5B-InstructvLLMQ5_K_S311 tok/s—1
Qwen2.5-0.5B-InstructStrataQ3_K_M294.8 tok/s—1
Qwen2.5-0.5B-InstructStrataQ5_K_M384.8 tok/s—1
Qwen2.5-Omni-3Bllama.cppIQ4_XS183.6 tok/s—1
Qwen2.5-Omni-3Bllama.cppQ2_K223.8 tok/s—1
Qwen2.5-Omni-3BOllamaIQ2_XS228.6 tok/s—1
Qwen2.5-Omni-3BOllamaIQ4_XS178.3 tok/s—1
Qwen3-Embedding-0.6Bllama.cppIQ2_XS354.8 tok/s—1
Qwen3-Embedding-0.6Bllama.cppIQ4_XS342.4 tok/s—1
gemma-4-12B-itvLLMIQ2_M125.4 tok/s—1
DeepSeek-R1-Distill-Qwen-14BvLLMIQ3_S96.6 tok/s—1
DeepSeek-R1-Distill-Qwen-14BvLLMQ5_K_M71.2 tok/s—1
DeepSeek-R1-Distill-Qwen-14BStrataIQ3_XS151.5 tok/s—1