0
Quantized builds
83
Benchmark records
1
Discussions
Quantized builds
Community and official quantized releases; "linked records" counts only benchmarks tied to that exact release.
No quantized builds cataloged yet.
Benchmark data
50 configurations
| Hardware | Framework | Quant | Decode | VRAM used | Samples | Evidence |
|---|---|---|---|---|---|---|
| NVIDIA RTX 4090 | llama.cpp | Q3_K_M | 122.1 tok/s | 22.09 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 3090 | llama.cpp | IQ4_XS | 79.3 tok/s | 22.8 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 3090 | llama.cpp | Q3_K_M | 75.7 tok/s | 22.09 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 3090 | vLLM | Q3_K_M | 80.1 tok/s | 22.09 GB | 1 | L2 Cross-framework verified |
| AMD Radeon RX 7900 XTX | llama.cpp | Q4_K_M | 65.8 tok/s | 22.8 GB | 1 | L2 Cross-framework verified |
| AMD Radeon RX 7900 XTX | Ollama | IQ2_XS | 90.8 tok/s | 16.21 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 5090 | vLLM | Q8_0 | 145.2 tok/s | 30.4 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 5080 | vLLM | IQ2_XS | 227.3 tok/s | 15.2 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 4080 Super | llama.cpp | IQ1_S | 208.1 tok/s | 14.45 GB | 1 | L2 Cross-framework verified |
| AMD Ryzen AI Max+ 395 | vLLM | Q3_K_S | 34.8 tok/s | 21.21 GB | 1 | L2 Cross-framework verified |
| AMD Ryzen AI Max+ 395 | vLLM | Q4_K_S | 29.6 tok/s | 25.03 GB | 1 | L2 Cross-framework verified |
| AMD Ryzen AI Max+ 395 | Ollama | Q4_K_S | 28.2 tok/s | 25.03 GB | 1 | L2 Cross-framework verified |
| Apple M5 Max | llama.cpp | Q3_K_M | 66.6 tok/s | 22.09 GB | 1 | L2 Cross-framework verified |
| Apple M5 Max | Ollama | IQ4_XS | 63.1 tok/s | 23.86 GB | 1 | L2 Cross-framework verified |
| Apple M5 Max | Ollama | Q8_0 | 36.8 tok/s | 35.91 GB | 1 | L2 Cross-framework verified |
| Apple M5 Ultra | llama.cpp | Q5_K_M | 77.6 tok/s | 29.44 GB | 1 | L2 Cross-framework verified |
| Apple M5 Ultra | Ollama | Q2_K | 118.3 tok/s | 18.27 GB | 1 | L2 Cross-framework verified |
| Apple M5 Ultra | Ollama | Q5_K_M | 65.7 tok/s | 29.44 GB | 1 | L2 Cross-framework verified |
| NVIDIA H100 80GB HBM3 | vLLM | IQ4_XS | 398.2 tok/s | 23.86 GB | 1 | L2 Cross-framework verified |
| NVIDIA H100 80GB HBM3 | vLLM | Q5_K_S | 364.8 tok/s | 28.56 GB | 1 | L2 Cross-framework verified |
| NVIDIA H100 80GB HBM3 | Ollama | IQ2_XS | 291.6 tok/s | 16.21 GB | 1 | L2 Cross-framework verified |
| NVIDIA H100 80GB HBM3 | Ollama | IQ3_XS | 340.5 tok/s | 20.33 GB | 1 | L2 Cross-framework verified |
| NVIDIA H20 | llama.cpp | IQ2_XS | 68 tok/s | 16.21 GB | 1 | L2 Cross-framework verified |
| NVIDIA H20 | llama.cpp | Q4_K_M | 42.4 tok/s | 26.21 GB | 1 | L2 Cross-framework verified |
| NVIDIA H20 | vLLM | Q2_K | 62.7 tok/s | 18.27 GB | 1 | L2 Cross-framework verified |
| NVIDIA H20 | vLLM | Q6_K | 33.5 tok/s | 31.5 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX PRO 6000 Blackwell 96GB | llama.cpp | Q5_K_S | 172.8 tok/s | 28.56 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX PRO 6000 Blackwell 96GB | vLLM | IQ3_S | 236.2 tok/s | 21.5 GB | 1 | L2 Cross-framework verified |
| Apple M2 Ultra | llama.cpp | Q4_K_S | 92.3 tok/s | 25.03 GB | 1 | L2 Cross-framework verified |
| Apple M2 Ultra | Ollama | Q3_K_S | 106.3 tok/s | 21.21 GB | 1 | L2 Cross-framework verified |
| Apple M2 Ultra | Ollama | Q5_K_M | 66.3 tok/s | 29.44 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX Spark | llama.cpp | Q4_K_M | 32.9 tok/s | 26.21 GB | 2 | L2 Cross-framework verified |
| NVIDIA RTX Spark | Ollama | IQ3_S | 37 tok/s | 21.5 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX Spark | Ollama | Q4_K_M | 32.3 tok/s | 26.21 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 4090 | vLLM | IQ2_M | 163.7 tok/s | 17.39 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 4090 | Ollama | Q3_K_L | 117.3 tok/s | 22.8 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 3090 | vLLM | Q2_K | 95.9 tok/s | 18.27 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 3090 | Ollama | IQ4_XS | 73.4 tok/s | 22.8 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 4060 Ti 16GB | vLLM | IQ2_XS | 125.3 tok/s | 15.2 GB | 1 | L2 Cross-framework verified |
| AMD Radeon RX 7900 XTX | llama.cpp | Q3_K_L | 85.5 tok/s | 22.8 GB | 1 | L2 Cross-framework verified |
| AMD Radeon RX 7900 XTX | vLLM | IQ2_M | 102.6 tok/s | 17.39 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 5090 | llama.cpp | IQ1_S | 313.2 tok/s | 14.45 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 5090 | vLLM | IQ3_XS | 290.3 tok/s | 20.33 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 5090 | vLLM | Q5_K_S | 193.3 tok/s | 28.56 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 5080 | llama.cpp | IQ2_XS | 202.8 tok/s | 15.2 GB | 1 | L2 Cross-framework verified |
| AMD Radeon RX 9070 XT | llama.cpp | IQ2_XS | 150.9 tok/s | 15.2 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 4080 Super | Strata | IQ2_M | 252.2 tok/s | 15.2 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 4080 | Ollama | IQ2_M | 163.6 tok/s | 15.2 GB | 1 | L2 Cross-framework verified |
| AMD Radeon RX 9070 XT | Ollama | IQ1_S | 91.3 tok/s | 14.45 GB | 1 | L2 Cross-framework verified |
| AMD Radeon RX 9070 | vLLM | Q2_K | 87.7 tok/s | 15.2 GB | 1 | L2 Cross-framework verified |
83 of these records are not linked to a specific quantized build; the full set is on the search page.