0
Quantized builds
34
Benchmark records
1
Discussions
Quantized builds
Community and official quantized releases; "linked records" counts only benchmarks tied to that exact release.
No quantized builds cataloged yet.
Benchmark data
34 configurations
| Hardware | Framework | Quant | Decode | VRAM used | Samples | Evidence |
|---|---|---|---|---|---|---|
| NVIDIA RTX Spark | llama.cpp | Q5_K_M | 78.8 tok/s | 70.45 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX Spark | llama.cpp | Q8_0 | 57.8 tok/s | 86.62 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX Spark | Ollama | Q8_0 | 52.4 tok/s | 86.62 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 5090 | Ollama | IQ1_S | 346.5 tok/s | 30.4 GB | 1 | L2 Cross-framework verified |
| AMD Ryzen AI Max+ 395 | vLLM | Q3_K_S | 72.5 tok/s | 49.88 GB | 1 | L2 Cross-framework verified |
| AMD Ryzen AI Max+ 395 | vLLM | Q6_K | 44.3 tok/s | 75.6 GB | 1 | L2 Cross-framework verified |
| AMD Ryzen AI Max+ 395 | Ollama | Q5_K_M | 45.6 tok/s | 70.45 GB | 1 | L2 Cross-framework verified |
| Apple M5 Max | llama.cpp | IQ3_XS | 159.3 tok/s | 47.67 GB | 1 | L2 Cross-framework verified |
| Apple M5 Max | llama.cpp | Q6_K | 90.4 tok/s | 75.6 GB | 1 | L2 Cross-framework verified |
| Apple M5 Max | Ollama | Q3_K_L | 131.3 tok/s | 55.02 GB | 1 | L2 Cross-framework verified |
| Apple M5 Max | Ollama | Q5_K_S | 105 tok/s | 68.25 GB | 1 | L2 Cross-framework verified |
| Apple M5 Ultra | llama.cpp | Q5_K_S | 169.2 tok/s | 68.25 GB | 1 | L2 Cross-framework verified |
| Apple M5 Ultra | Ollama | Q3_K_M | 206.7 tok/s | 52.08 GB | 1 | L2 Cross-framework verified |
| NVIDIA H100 80GB HBM3 | vLLM | IQ4_XS | 288.7 tok/s | 56.49 GB | 1 | L2 Cross-framework verified |
| NVIDIA H100 80GB HBM3 | vLLM | Q3_K_M | 355.6 tok/s | 52.08 GB | 1 | L2 Cross-framework verified |
| NVIDIA H100 80GB HBM3 | Ollama | IQ2_M | 321.9 tok/s | 40.32 GB | 1 | L2 Cross-framework verified |
| NVIDIA H100 80GB HBM3 | Ollama | Q8_0 | 296.9 tok/s | 76 GB | 1 | L2 Cross-framework verified |
| NVIDIA H20 | vLLM | Q3_K_S | 118.5 tok/s | 49.88 GB | 1 | L2 Cross-framework verified |
| NVIDIA H20 | vLLM | Q8_0 | 66.6 tok/s | 86.62 GB | 1 | L2 Cross-framework verified |
| NVIDIA H20 | Ollama | Q3_K_M | 90.2 tok/s | 52.08 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX PRO 6000 Blackwell 96GB | vLLM | IQ3_XS | 319.3 tok/s | 47.67 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX PRO 6000 Blackwell 96GB | Ollama | Q5_K_S | 301.4 tok/s | 68.25 GB | 1 | L2 Cross-framework verified |
| Apple M2 Ultra | llama.cpp | IQ1_S | 250.5 tok/s | 32.97 GB | 1 | L2 Cross-framework verified |
| Apple M2 Ultra | llama.cpp | Q8_0 | 115 tok/s | 76 GB | 1 | L2 Cross-framework verified |
| Apple M2 Ultra | Ollama | Q2_K | 217.1 tok/s | 42.52 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX Spark | llama.cpp | IQ3_XS | 117.7 tok/s | 47.67 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX Spark | llama.cpp | Q4_K_M | 88.7 tok/s | 58.9 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX Spark | Ollama | IQ2_XS | 128.3 tok/s | 37.38 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX Spark | Ollama | Q4_K_M | 85.4 tok/s | 58.9 GB | 1 | L2 Cross-framework verified |
| Apple M3 Max | llama.cpp | IQ2_XS | 89.4 tok/s | 34.2 GB | 1 | L2 Cross-framework verified |
| Apple M3 Max | vLLM | IQ1_S | 102.6 tok/s | 32.97 GB | 1 | L2 Cross-framework verified |
| Apple M4 Max | llama.cpp | IQ3_XS | 121.6 tok/s | 47.67 GB | 1 | L2 Cross-framework verified |
| Apple M4 Max | llama.cpp | Q4_K_S | 111 tok/s | 59.43 GB | 1 | L2 Cross-framework verified |
| Apple M4 Max | Ollama | Q3_K_L | 102.3 tok/s | 55.02 GB | 1 | L2 Cross-framework verified |
34 of these records are not linked to a specific quantized build; the full set is on the search page.