0
Quantized builds
90
Benchmark records
Quantized builds
Community and official quantized releases; "linked records" counts only benchmarks tied to that exact release.
No quantized builds cataloged yet.
Benchmark data
50 configurations
| Hardware | Framework | Quant | Decode | VRAM used | Samples | Evidence |
|---|---|---|---|---|---|---|
| NVIDIA RTX 5090 | llama.cpp | FP16 | 149.3 tok/s | 17.64 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 5090 | llama.cpp | IQ4_XS | 335.8 tok/s | 8.32 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 5090 | Ollama | IQ3_XS | 349.8 tok/s | 7.31 GB | 1 | L2 Cross-framework verified |
| AMD Ryzen AI Max+ 395 | llama.cpp | Q6_K | 104.5 tok/s | 10.5 GB | 1 | L2 Cross-framework verified |
| AMD Ryzen AI Max+ 395 | Ollama | IQ1_S | 170.9 tok/s | 5.63 GB | 1 | L2 Cross-framework verified |
| AMD Ryzen AI Max+ 395 | Ollama | IQ2_M | 170.8 tok/s | 6.47 GB | 1 | L2 Cross-framework verified |
| Apple M5 Max | llama.cpp | Q3_K_L | 253.6 tok/s | 8.15 GB | 1 | L2 Cross-framework verified |
| Apple M5 Max | Ollama | IQ2_XS | 308.4 tok/s | 6.13 GB | 2 | L2 Cross-framework verified |
| Apple M5 Max | Ollama | IQ3_XS | 249.1 tok/s | 7.31 GB | 1 | L2 Cross-framework verified |
| NVIDIA DGX Spark | llama.cpp | Q3_K_M | 206.5 tok/s | 7.81 GB | 1 | L2 Cross-framework verified |
| NVIDIA DGX Spark | Strata | Q3_K_S | 338 tok/s | 7.56 GB | 1 | L2 Cross-framework verified |
| NVIDIA DGX Spark | Strata | Q5_K_M | 221 tok/s | 9.91 GB | 2 | L2 Cross-framework verified |
| NVIDIA Jetson AGX Orin 64GB | llama.cpp | IQ2_M | 92.8 tok/s | 6.47 GB | 1 | L2 Cross-framework verified |
| NVIDIA Jetson AGX Orin 64GB | vLLM | IQ2_M | 96.3 tok/s | 6.47 GB | 1 | L2 Cross-framework verified |
| NVIDIA Jetson AGX Orin 64GB | vLLM | Q2_K | 95.8 tok/s | 6.72 GB | 1 | L2 Cross-framework verified |
| AMD Strix Halo (Radeon 8060S) | llama.cpp | Q4_K_S | 106.7 tok/s | 8.65 GB | 1 | L2 Cross-framework verified |
| AMD Strix Halo (Radeon 8060S) | Strata | Q4_K_S | 150.6 tok/s | 8.65 GB | 1 | L2 Cross-framework verified |
| AMD Strix Halo (Radeon 8060S) | Strata | Q6_K | 108.8 tok/s | 10.5 GB | 1 | L2 Cross-framework verified |
| 2x AMD Strix Halo | llama.cpp | FP16 | 87 tok/s | 17.64 GB | 1 | L2 Cross-framework verified |
| 2x AMD Strix Halo | Ollama | IQ3_S | 197.2 tok/s | 7.64 GB | 1 | L2 Cross-framework verified |
| 2x AMD Strix Halo | Ollama | Q4_K_M | 165.7 tok/s | 8.99 GB | 1 | L2 Cross-framework verified |
| AMD Strix Point (Radeon 890M) | vLLM | FP16 | 36.1 tok/s | 15.2 GB | 1 | L2 Cross-framework verified |
| AMD Strix Point (Radeon 890M) | vLLM | IQ2_XS | 112.8 tok/s | 6.13 GB | 1 | L2 Cross-framework verified |
| AMD Strix Point (Radeon 890M) | Ollama | IQ4_XS | 68.6 tok/s | 8.32 GB | 1 | L2 Cross-framework verified |
| AMD Strix Point (Radeon 890M) | Ollama | Q3_K_L | 75.1 tok/s | 8.15 GB | 1 | L2 Cross-framework verified |
| Intel Arc A770 16GB | vLLM | IQ4_XS | 72 tok/s | 8.32 GB | 1 | L2 Cross-framework verified |
| Intel Arc A770 16GB | Strata | Q6_K | 68.9 tok/s | 10.5 GB | 1 | L2 Cross-framework verified |
| Intel Arc 140V | vLLM | IQ3_S | 80.1 tok/s | 7.64 GB | 1 | L2 Cross-framework verified |
| Intel Arc 140V | vLLM | Q8_0 | 42.9 tok/s | 11.76 GB | 1 | L2 Cross-framework verified |
| Intel Arc 140V | Ollama | IQ1_S | 81.3 tok/s | 5.63 GB | 1 | L2 Cross-framework verified |
| Intel Arc 140V | Ollama | Q4_K_M | 54.5 tok/s | 8.99 GB | 1 | L2 Cross-framework verified |
| Apple M1 | llama.cpp | Q3_K_L | 65.6 tok/s | 8.15 GB | 1 | L2 Cross-framework verified |
| Apple M1 | llama.cpp | Q8_0 | 36.8 tok/s | 11.76 GB | 1 | L2 Cross-framework verified |
| Apple M1 | Ollama | Q2_K | 72.5 tok/s | 6.72 GB | 1 | L2 Cross-framework verified |
| Apple M1 Pro | Ollama | IQ3_XS | 98.6 tok/s | 7.31 GB | 1 | L2 Cross-framework verified |
| Apple M1 Pro | Strata | FP16 | 58.7 tok/s | 17.64 GB | 1 | L2 Cross-framework verified |
| Apple M1 Max | llama.cpp | Q2_K | 145.2 tok/s | 6.72 GB | 1 | L2 Cross-framework verified |
| Apple M1 Max | Ollama | IQ2_M | 126.4 tok/s | 6.47 GB | 1 | L2 Cross-framework verified |
| Apple M1 Max | Ollama | Q8_0 | 64.7 tok/s | 11.76 GB | 1 | L2 Cross-framework verified |
| Apple M2 Pro | vLLM | FP16 | 48.5 tok/s | 17.64 GB | 1 | L2 Cross-framework verified |
| Apple M2 Pro | vLLM | Q6_K | 73 tok/s | 10.5 GB | 1 | L2 Cross-framework verified |
| Apple M2 Pro | Strata | FP16 | 65.2 tok/s | 17.64 GB | 1 | L2 Cross-framework verified |
| Apple M2 Ultra | Ollama | FP16 | 64.1 tok/s | 17.64 GB | 1 | L2 Cross-framework verified |
| Apple M2 Ultra | Ollama | IQ2_XS | 161.9 tok/s | 6.13 GB | 1 | L2 Cross-framework verified |
| Apple M2 Ultra | Strata | Q3_K_L | 200.6 tok/s | 8.15 GB | 1 | L2 Cross-framework verified |
| Apple M2 Ultra | Strata | Q5_K_S | 162.1 tok/s | 9.66 GB | 1 | L2 Cross-framework verified |
| Qualcomm Snapdragon X Elite | vLLM | IQ4_XS | 66.1 tok/s | 8.32 GB | 1 | L2 Cross-framework verified |
| Qualcomm Snapdragon X Elite | vLLM | Q3_K_M | 74.2 tok/s | 7.81 GB | 1 | L2 Cross-framework verified |
| Qualcomm Snapdragon X Elite | Strata | FP16 | 39.4 tok/s | 15.2 GB | 1 | L2 Cross-framework verified |
| Qualcomm Snapdragon 7 Gen 4 | Ollama | Q2_K | 58.8 tok/s | 6.72 GB | 1 | L2 Cross-framework verified |
90 of these records are not linked to a specific quantized build; the full set is on the search page.
Community discussion
No discussions yet — start the first one.
Start a discussion