0
Quantized builds
106
Benchmark records
1
Discussions
Quantized builds
Community and official quantized releases; "linked records" counts only benchmarks tied to that exact release.
No quantized builds cataloged yet.
Benchmark data
50 configurations
| Hardware | Framework | Quant | Decode | VRAM used | Samples | Evidence |
|---|---|---|---|---|---|---|
| NVIDIA RTX 4090 | vLLM | IQ3_S | 327.6 tok/s | 6.26 GB | 2 | L2 Cross-framework verified |
| NVIDIA RTX 4090 | Ollama | Q4_K_S | 336.8 tok/s | 7.01 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 4090 | Ollama | Q5_K_M | 263.1 tok/s | 7.96 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 3090 | llama.cpp | Q2_K | 287.4 tok/s | 5.57 GB | 2 | L2 Cross-framework verified |
| NVIDIA RTX 3090 | vLLM | FP16 | 114 tok/s | 13.76 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 3090 | vLLM | Q3_K_M | 250.5 tok/s | 6.38 GB | 1 | L2 Cross-framework verified |
| AMD Radeon RX 7900 XTX | vLLM | Q6_K | 175.9 tok/s | 8.4 GB | 1 | L2 Cross-framework verified |
| AMD Radeon RX 7900 XTX | Ollama | Q3_K_L | 237.9 tok/s | 6.64 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 5090 | llama.cpp | Q4_K_S | 375.6 tok/s | 7.01 GB | 2 | L2 Cross-framework verified |
| NVIDIA RTX 5090 | llama.cpp | Q5_K_S | 373.6 tok/s | 7.77 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 5090 | vLLM | IQ4_XS | 321.2 tok/s | 6.76 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 5090 | vLLM | Q6_K | 292.4 tok/s | 8.4 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 5080 | llama.cpp | Q5_K_S | 381.8 tok/s | 7.77 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 5080 | Ollama | Q5_K_S | 312.2 tok/s | 7.77 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 5080 | Ollama | Q8_0 | 294 tok/s | 9.35 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 3060 12GB | vLLM | Q4_K_S | 182.9 tok/s | 7.01 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 3060 12GB | vLLM | Q5_K_S | 145 tok/s | 7.77 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 3060 12GB | Strata | Q6_K | 168.1 tok/s | 8.4 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 4080 Super | llama.cpp | IQ2_XS | 373 tok/s | 5.12 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 4080 Super | llama.cpp | IQ4_XS | 290 tok/s | 6.76 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 4080 Super | Strata | FP16 | 277.1 tok/s | 13.76 GB | 1 | L2 Cross-framework verified |
| AMD Ryzen AI Max+ 395 | llama.cpp | Q8_0 | 61.6 tok/s | 9.35 GB | 1 | L2 Cross-framework verified |
| AMD Ryzen AI Max+ 395 | vLLM | Q4_K_M | 89.3 tok/s | 7.27 GB | 1 | L2 Cross-framework verified |
| Apple M5 Max | llama.cpp | Q4_K_S | 188.3 tok/s | 7.01 GB | 1 | L2 Cross-framework verified |
| Apple M5 Max | Ollama | IQ3_XS | 209.8 tok/s | 6.01 GB | 1 | L2 Cross-framework verified |
| Apple M5 Ultra | llama.cpp | Q3_K_S | 338.3 tok/s | 6.2 GB | 1 | L2 Cross-framework verified |
| Apple M5 Ultra | Ollama | Q3_K_S | 318.4 tok/s | 6.2 GB | 1 | L2 Cross-framework verified |
| NVIDIA H100 80GB HBM3 | llama.cpp | FP16 | 379.9 tok/s | 13.76 GB | 1 | L2 Cross-framework verified |
| NVIDIA H100 80GB HBM3 | llama.cpp | Q5_K_M | 313.8 tok/s | 7.96 GB | 1 | L2 Cross-framework verified |
| NVIDIA H100 80GB HBM3 | Ollama | FP16 | 308.8 tok/s | 13.76 GB | 1 | L2 Cross-framework verified |
| NVIDIA H100 80GB HBM3 | Ollama | IQ2_XS | 295.4 tok/s | 5.12 GB | 1 | L2 Cross-framework verified |
| NVIDIA H20 | llama.cpp | Q3_K_S | 171.3 tok/s | 6.2 GB | 1 | L2 Cross-framework verified |
| NVIDIA H20 | vLLM | FP16 | 76.2 tok/s | 13.76 GB | 1 | L2 Cross-framework verified |
| NVIDIA H20 | vLLM | IQ3_XS | 177.1 tok/s | 6.01 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX PRO 6000 Blackwell 96GB | llama.cpp | IQ3_XS | 338 tok/s | 6.01 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX PRO 6000 Blackwell 96GB | llama.cpp | Q8_0 | 331.1 tok/s | 9.35 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX PRO 6000 Blackwell 96GB | Ollama | IQ3_XS | 283.4 tok/s | 6.01 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX PRO 6000 Blackwell 96GB | Ollama | Q3_K_S | 364.1 tok/s | 6.2 GB | 1 | L2 Cross-framework verified |
| Intel Arc B580 | llama.cpp | Q2_K | 169.6 tok/s | 5.57 GB | 1 | L2 Cross-framework verified |
| Intel Arc B580 | llama.cpp | Q5_K_S | 120.7 tok/s | 7.77 GB | 1 | L2 Cross-framework verified |
| Intel Arc B580 | Ollama | FP16 | 62.5 tok/s | 11.4 GB | 1 | L2 Cross-framework verified |
| Apple M2 Ultra | llama.cpp | Q5_K_M | 233.8 tok/s | 7.96 GB | 1 | L2 Cross-framework verified |
| Apple M2 Ultra | Ollama | Q5_K_M | 207 tok/s | 7.96 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX Spark | llama.cpp | IQ3_S | 108.8 tok/s | 6.26 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX Spark | llama.cpp | Q4_K_M | 89.5 tok/s | 7.27 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX Spark | Ollama | IQ2_XS | 115.7 tok/s | 5.12 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX Spark | Ollama | Q4_K_M | 76.6 tok/s | 7.27 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 4090 | Strata | IQ1_S | 284.1 tok/s | 4.75 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 4090 | Strata | IQ2_XS | 281.2 tok/s | 5.12 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 3090 | Strata | FP16 | 145.3 tok/s | 13.76 GB | 1 | L2 Cross-framework verified |
106 of these records are not linked to a specific quantized build; the full set is on the search page.