0
Quantized builds
88
Benchmark records
Quantized builds
Community and official quantized releases; "linked records" counts only benchmarks tied to that exact release.
No quantized builds cataloged yet.
Benchmark data
50 configurations
| Hardware | Framework | Quant | Decode | VRAM used | Samples | Evidence |
|---|---|---|---|---|---|---|
| NVIDIA RTX 5090 | llama.cpp | IQ4_XS | 319.1 tok/s | 2.88 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 5090 | llama.cpp | Q8_0 | 359 tok/s | 3.31 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 5090 | vLLM | IQ3_XS | 285.4 tok/s | 2.75 GB | 1 | L2 Cross-framework verified |
| AMD Ryzen AI Max+ 395 | llama.cpp | Q3_K_M | 339.2 tok/s | 2.81 GB | 1 | L2 Cross-framework verified |
| AMD Ryzen AI Max+ 395 | llama.cpp | Q3_K_S | 367 tok/s | 2.78 GB | 1 | L2 Cross-framework verified |
| AMD Ryzen AI Max+ 395 | Strata | IQ2_M | 347.2 tok/s | 2.65 GB | 1 | L2 Cross-framework verified |
| Apple M5 Max | vLLM | IQ2_XS | 291.8 tok/s | 2.6 GB | 1 | L2 Cross-framework verified |
| Apple M5 Max | Strata | IQ1_S | 343.9 tok/s | 2.54 GB | 1 | L2 Cross-framework verified |
| Apple M5 Max | Strata | Q5_K_S | 305.2 tok/s | 3.04 GB | 1 | L2 Cross-framework verified |
| NVIDIA DGX Spark | llama.cpp | IQ2_XS | 344.7 tok/s | 2.6 GB | 1 | L2 Cross-framework verified |
| NVIDIA DGX Spark | Strata | IQ2_XS | 295.8 tok/s | 2.6 GB | 1 | L2 Cross-framework verified |
| NVIDIA DGX Spark | Strata | Q5_K_M | 294.1 tok/s | 3.08 GB | 1 | L2 Cross-framework verified |
| NVIDIA Jetson AGX Orin 64GB | llama.cpp | IQ1_S | 243.5 tok/s | 2.54 GB | 1 | L2 Cross-framework verified |
| NVIDIA Jetson AGX Orin 64GB | llama.cpp | Q3_K_S | 199.9 tok/s | 2.78 GB | 1 | L2 Cross-framework verified |
| NVIDIA Jetson AGX Orin 64GB | Strata | IQ3_XS | 310.9 tok/s | 2.75 GB | 1 | L2 Cross-framework verified |
| AMD Strix Halo (Radeon 8060S) | llama.cpp | FP16 | 124.5 tok/s | 4.04 GB | 1 | L2 Cross-framework verified |
| AMD Strix Halo (Radeon 8060S) | llama.cpp | Q6_K | 171.4 tok/s | 3.15 GB | 1 | L2 Cross-framework verified |
| AMD Strix Halo (Radeon 8060S) | vLLM | Q5_K_S | 228.5 tok/s | 3.04 GB | 1 | L2 Cross-framework verified |
| 2x AMD Strix Halo | Ollama | Q3_K_S | 347.5 tok/s | 2.78 GB | 1 | L2 Cross-framework verified |
| 2x AMD Strix Halo | Ollama | Q8_0 | 257 tok/s | 3.31 GB | 1 | L2 Cross-framework verified |
| 2x AMD Strix Halo | Strata | Q4_K_M | 303 tok/s | 2.96 GB | 1 | L2 Cross-framework verified |
| AMD Strix Point (Radeon 890M) | llama.cpp | Q4_K_S | 160.4 tok/s | 2.92 GB | 1 | L2 Cross-framework verified |
| AMD Strix Point (Radeon 890M) | Strata | Q4_K_S | 231.1 tok/s | 2.92 GB | 1 | L2 Cross-framework verified |
| Intel Arc A770 16GB | llama.cpp | IQ3_XS | 195.3 tok/s | 2.75 GB | 1 | L2 Cross-framework verified |
| Intel Arc A770 16GB | llama.cpp | IQ4_XS | 170 tok/s | 2.88 GB | 1 | L2 Cross-framework verified |
| Intel Arc A770 16GB | vLLM | IQ1_S | 232.6 tok/s | 2.54 GB | 1 | L2 Cross-framework verified |
| Intel Arc A770 16GB | vLLM | IQ2_M | 228.6 tok/s | 2.65 GB | 1 | L2 Cross-framework verified |
| Intel Arc 140V | Ollama | IQ1_S | 197.4 tok/s | 2.54 GB | 1 | L2 Cross-framework verified |
| Intel Arc 140V | Ollama | Q3_K_L | 140.9 tok/s | 2.86 GB | 1 | L2 Cross-framework verified |
| Intel Arc 140V | Strata | Q2_K | 267.4 tok/s | 2.68 GB | 1 | L2 Cross-framework verified |
| Intel Arc 140V | Strata | Q6_K | 163.3 tok/s | 3.15 GB | 1 | L2 Cross-framework verified |
| Apple M1 | llama.cpp | IQ2_M | 189.8 tok/s | 2.65 GB | 1 | L2 Cross-framework verified |
| Apple M1 | llama.cpp | Q2_K | 189.4 tok/s | 2.68 GB | 1 | L2 Cross-framework verified |
| Apple M1 | vLLM | Q3_K_S | 170.3 tok/s | 2.78 GB | 1 | L2 Cross-framework verified |
| Apple M1 | vLLM | Q4_K_M | 144 tok/s | 2.96 GB | 1 | L2 Cross-framework verified |
| Apple M1 Pro | llama.cpp | Q4_K_M | 190.8 tok/s | 2.96 GB | 1 | L2 Cross-framework verified |
| Apple M1 Pro | vLLM | Q4_K_M | 203.2 tok/s | 2.96 GB | 1 | L2 Cross-framework verified |
| Apple M1 Max | llama.cpp | Q4_K_S | 266.4 tok/s | 2.92 GB | 1 | L2 Cross-framework verified |
| Apple M1 Max | Strata | Q3_K_L | 395.3 tok/s | 2.86 GB | 1 | L2 Cross-framework verified |
| Apple M2 Pro | vLLM | Q3_K_L | 248.2 tok/s | 2.86 GB | 1 | L2 Cross-framework verified |
| Apple M2 Pro | Ollama | Q5_K_S | 165.8 tok/s | 3.04 GB | 1 | L2 Cross-framework verified |
| Apple M2 Pro | Ollama | Q8_0 | 140.5 tok/s | 3.31 GB | 1 | L2 Cross-framework verified |
| Apple M2 Ultra | vLLM | Q4_K_S | 318.4 tok/s | 2.92 GB | 1 | L2 Cross-framework verified |
| Apple M2 Ultra | Ollama | Q2_K | 393.9 tok/s | 2.68 GB | 1 | L2 Cross-framework verified |
| Apple M2 Ultra | Ollama | Q4_K_M | 265.6 tok/s | 2.96 GB | 1 | L2 Cross-framework verified |
| Qualcomm Snapdragon X Elite | Ollama | IQ4_XS | 131.7 tok/s | 2.88 GB | 1 | L2 Cross-framework verified |
| Qualcomm Snapdragon X Elite | Strata | FP16 | 96.3 tok/s | 4.04 GB | 1 | L2 Cross-framework verified |
| Qualcomm Snapdragon 7 Gen 4 | vLLM | IQ3_XS | 150.8 tok/s | 2.75 GB | 1 | L2 Cross-framework verified |
| Qualcomm Snapdragon 7 Gen 4 | Strata | Q6_K | 115.5 tok/s | 3.15 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 4060 Ti 16GB | llama.cpp | IQ1_S | 337.1 tok/s | 2.54 GB | 1 | L2 Cross-framework verified |
88 of these records are not linked to a specific quantized build; the full set is on the search page.
Community discussion
No discussions yet — start the first one.
Start a discussion