AMD Radeon AI PRO R9700
GPU33
Run records
13
Models covered
4
Frameworks covered
L2 Cross-framework verified × 33
Evidence mix
Typical measured performance
Published benchmark averages per model × framework × quantization on this hardware.
- llama.cpp · Q3_K_M Decode 183.89 · Prefill 727.67 ·
- vLLM · IQ3_XS Decode 208.55 · Prefill 738.48 ·
- vLLM · Q5_K_M Decode 122.86 · Prefill 611.36 ·
- vLLM · Q4_K_M Decode 98.86 · Prefill 492.64 ·
- vLLM · Q6_K Decode 76.56 · Prefill 349.76 ·
- llama.cpp · IQ2_XS Decode 394.48 · Prefill 1743.78 ·
- Ollama · IQ1_S Decode 368.92 · Prefill 1396.86 ·
- Ollama · Q4_K_S Decode 284.75 · Prefill 1200.05 ·
- Ollama · Q3_K_M Decode 123.24 · Prefill 659.52 ·
- Ollama · Q3_K_S Decode 110.64 · Prefill 546.12 ·
- llama.cpp · IQ2_M Decode 154.04 · Prefill 793.94 ·
- llama.cpp · Q3_K_S Decode 137.81 · Prefill 521.21 ·
- Strata · IQ1_S Decode 246.4 · Prefill 1052.77 ·
- Strata · IQ3_XS Decode 190.7 · Prefill 730.63 ·
- Strata · Q4_K_S Decode 204.13 · Prefill 1042.53 ·
- vLLM · Q3_K_S Decode 272.77 · Prefill 1363 ·
- Ollama · IQ4_XS Decode 192.57 · Prefill 691.62 ·
- vLLM · Q4_K_S Decode 158.57 · Prefill 583.28 ·
- vLLM · Q5_K_M Decode 111.48 · Prefill 571.17 ·
- vLLM · IQ3_XS Decode 346.96 · Prefill 1652.99 ·
- vLLM · Q5_K_S Decode 310.96 · Prefill 1151.48 ·
- Strata · Q3_K_M Decode 294.81 · Prefill 1272.19 ·
- Strata · Q5_K_M Decode 384.75 · Prefill 2096.67 ·
- llama.cpp · IQ4_XS Decode 183.61 · Prefill 748.24 ·
- llama.cpp · Q2_K Decode 223.75 · Prefill 955.8 ·
- Ollama · IQ2_XS Decode 228.57 · Prefill 1151.43 ·
- Ollama · IQ4_XS Decode 178.25 · Prefill 834.26 ·
- llama.cpp · IQ2_XS Decode 354.83 · Prefill 1505.4 ·
- llama.cpp · IQ4_XS Decode 342.35 · Prefill 1387.76 ·
- vLLM · IQ2_M Decode 125.41 · Prefill 606.9 ·
- vLLM · IQ3_S Decode 96.64 · Prefill 451.11 ·
- vLLM · Q5_K_M Decode 71.18 · Prefill 263.48 ·
- Strata · IQ3_XS Decode 151.48 · Prefill 693.94 ·
Fit check
Physics-based estimate from VRAM and memory bandwidth: pick a model to see which quantizations fit and the expected speed
Formula: tok/s = (bandwidth ÷ model size) × 0.55, adapted from llmfit; see run records for measured numbers
Key specs
- Vendor
- AMD
- VRAM
- 32 GB
- Memory bandwidth
- 512 GB/s
Run records
33 configurations
| Model | Framework | Quantization | Decode | Gen time | Samples |
|---|---|---|---|---|---|
| Qwen3.5-4B | llama.cpp | Q3_K_M | 183.9 tok/s | — | 1 |
| Qwen3.5-4B | vLLM | IQ3_XS | 208.6 tok/s | — | 1 |
| Qwen3.5-4B | vLLM | Q5_K_M | 122.9 tok/s | — | 1 |
| Qwen3.5-9B | vLLM | Q4_K_M | 98.9 tok/s | — | 1 |
| Qwen3.5-9B | vLLM | Q6_K | 76.6 tok/s | — | 1 |
| Qwen3.5-0.8B | llama.cpp | IQ2_XS | 394.5 tok/s | — | 1 |
| Qwen3.5-0.8B | Ollama | IQ1_S | 368.9 tok/s | — | 1 |
| Qwen3.5-0.8B | Ollama | Q4_K_S | 284.8 tok/s | — | 1 |
| Qwen3.8-9B | Ollama | Q3_K_M | 123.2 tok/s | — | 1 |
| Qwen3.8-9B | Ollama | Q3_K_S | 110.6 tok/s | — | 1 |
| Qwen3-8B | llama.cpp | IQ2_M | 154 tok/s | — | 1 |
| Qwen3-8B | llama.cpp | Q3_K_S | 137.8 tok/s | — | 1 |
| Qwen3-8B | Strata | IQ1_S | 246.4 tok/s | — | 1 |
| Qwen3-8B | Strata | IQ3_XS | 190.7 tok/s | — | 1 |
| Qwen3-4B | Strata | Q4_K_S | 204.1 tok/s | — | 1 |
| Qwen3-1.7B | vLLM | Q3_K_S | 272.8 tok/s | — | 1 |
| Qwen3-1.7B | Ollama | IQ4_XS | 192.6 tok/s | — | 1 |
| Qwen3-4B-Thinking-2507 | vLLM | Q4_K_S | 158.6 tok/s | — | 1 |
| Qwen3-4B-Thinking-2507 | vLLM | Q5_K_M | 111.5 tok/s | — | 1 |
| Qwen2.5-0.5B-Instruct | vLLM | IQ3_XS | 347 tok/s | — | 1 |
| Qwen2.5-0.5B-Instruct | vLLM | Q5_K_S | 311 tok/s | — | 1 |
| Qwen2.5-0.5B-Instruct | Strata | Q3_K_M | 294.8 tok/s | — | 1 |
| Qwen2.5-0.5B-Instruct | Strata | Q5_K_M | 384.8 tok/s | — | 1 |
| Qwen2.5-Omni-3B | llama.cpp | IQ4_XS | 183.6 tok/s | — | 1 |
| Qwen2.5-Omni-3B | llama.cpp | Q2_K | 223.8 tok/s | — | 1 |
| Qwen2.5-Omni-3B | Ollama | IQ2_XS | 228.6 tok/s | — | 1 |
| Qwen2.5-Omni-3B | Ollama | IQ4_XS | 178.3 tok/s | — | 1 |
| Qwen3-Embedding-0.6B | llama.cpp | IQ2_XS | 354.8 tok/s | — | 1 |
| Qwen3-Embedding-0.6B | llama.cpp | IQ4_XS | 342.4 tok/s | — | 1 |
| gemma-4-12B-it | vLLM | IQ2_M | 125.4 tok/s | — | 1 |
| DeepSeek-R1-Distill-Qwen-14B | vLLM | IQ3_S | 96.6 tok/s | — | 1 |
| DeepSeek-R1-Distill-Qwen-14B | vLLM | Q5_K_M | 71.2 tok/s | — | 1 |
| DeepSeek-R1-Distill-Qwen-14B | Strata | IQ3_XS | 151.5 tok/s | — | 1 |