0
Quantized builds
91
Benchmark records
Quantized builds
Community and official quantized releases; "linked records" counts only benchmarks tied to that exact release.
No quantized builds cataloged yet.
Benchmark data
50 configurations
| Hardware | Framework | Quant | Decode | VRAM used | Samples | Evidence |
|---|---|---|---|---|---|---|
| NVIDIA RTX 4090 | llama.cpp | Q5_K_S | 118.7 tok/s | 17.22 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 4090 | vLLM | IQ3_S | 177.4 tok/s | 13.19 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 4090 | vLLM | Q4_K_S | 157.4 tok/s | 15.2 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 3090 | Ollama | Q5_K_S | 68 tok/s | 17.22 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 3090 | Strata | IQ3_S | 155.9 tok/s | 13.19 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 3090 | Strata | Q5_K_M | 101.9 tok/s | 17.72 GB | 1 | L2 Cross-framework verified |
| AMD Radeon RX 7900 XTX | vLLM | Q4_K_M | 95.7 tok/s | 15.88 GB | 1 | L2 Cross-framework verified |
| AMD Radeon RX 7900 XTX | Ollama | Q5_K_S | 74.7 tok/s | 17.22 GB | 2 | L2 Cross-framework verified |
| NVIDIA RTX 5090 | vLLM | IQ3_XS | 353.4 tok/s | 12.52 GB | 2 | L2 Cross-framework verified |
| NVIDIA RTX 5090 | vLLM | Q4_K_M | 288.4 tok/s | 15.88 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 5090 | Ollama | Q5_K_M | 226.3 tok/s | 17.72 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 5080 | Ollama | IQ3_S | 199.2 tok/s | 13.19 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 5080 | Ollama | Q3_K_S | 204.5 tok/s | 13.02 GB | 2 | L2 Cross-framework verified |
| NVIDIA RTX 5080 | Strata | IQ1_S | 349.7 tok/s | 9.16 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 5080 | Strata | Q3_K_L | 297.9 tok/s | 14.2 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 3060 12GB | Strata | Q3_K_M | 95.2 tok/s | 11.4 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 4080 Super | Ollama | IQ3_XS | 190.9 tok/s | 12.52 GB | 2 | L2 Cross-framework verified |
| NVIDIA RTX 4080 Super | Strata | IQ2_XS | 334.5 tok/s | 10.16 GB | 1 | L2 Cross-framework verified |
| AMD Ryzen AI Max+ 395 | vLLM | IQ2_M | 48.9 tok/s | 10.84 GB | 1 | L2 Cross-framework verified |
| AMD Ryzen AI Max+ 395 | vLLM | Q5_K_M | 31.8 tok/s | 17.72 GB | 1 | L2 Cross-framework verified |
| AMD Ryzen AI Max+ 395 | Ollama | Q3_K_L | 37.4 tok/s | 14.2 GB | 1 | L2 Cross-framework verified |
| Apple M5 Max | llama.cpp | Q3_K_M | 83.6 tok/s | 13.52 GB | 1 | L2 Cross-framework verified |
| Apple M5 Max | Ollama | IQ2_M | 100 tok/s | 10.84 GB | 1 | L2 Cross-framework verified |
| Apple M5 Ultra | llama.cpp | IQ2_M | 145.5 tok/s | 10.84 GB | 1 | L2 Cross-framework verified |
| Apple M5 Ultra | llama.cpp | Q4_K_S | 115.5 tok/s | 15.2 GB | 1 | L2 Cross-framework verified |
| Apple M5 Ultra | Ollama | IQ3_XS | 140.7 tok/s | 12.52 GB | 1 | L2 Cross-framework verified |
| NVIDIA H100 80GB HBM3 | llama.cpp | IQ3_S | 289.2 tok/s | 13.19 GB | 1 | L2 Cross-framework verified |
| NVIDIA H100 80GB HBM3 | llama.cpp | IQ4_XS | 283.2 tok/s | 14.53 GB | 1 | L2 Cross-framework verified |
| NVIDIA H100 80GB HBM3 | vLLM | Q2_K | 349.2 tok/s | 11.34 GB | 1 | L2 Cross-framework verified |
| NVIDIA H20 | vLLM | Q2_K | 77.7 tok/s | 11.34 GB | 1 | L2 Cross-framework verified |
| NVIDIA H20 | Ollama | Q5_K_M | 41.5 tok/s | 17.72 GB | 1 | L2 Cross-framework verified |
| NVIDIA H20 | Ollama | Q8_0 | 36.5 tok/s | 21.42 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX PRO 6000 Blackwell 96GB | llama.cpp | IQ2_M | 345.7 tok/s | 10.84 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX PRO 6000 Blackwell 96GB | llama.cpp | Q5_K_M | 194.4 tok/s | 17.72 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX PRO 6000 Blackwell 96GB | Ollama | Q3_K_M | 249.2 tok/s | 13.52 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX PRO 6000 Blackwell 96GB | Ollama | Q6_K | 185.7 tok/s | 18.9 GB | 1 | L2 Cross-framework verified |
| Intel Arc B580 | llama.cpp | Q3_K_M | 57.8 tok/s | 11.4 GB | 1 | L2 Cross-framework verified |
| Intel Arc B580 | Ollama | Q2_K | 67 tok/s | 11.34 GB | 1 | L2 Cross-framework verified |
| Apple M2 Ultra | llama.cpp | IQ2_M | 144 tok/s | 10.84 GB | 1 | L2 Cross-framework verified |
| Apple M2 Ultra | llama.cpp | Q2_K | 151.5 tok/s | 11.34 GB | 1 | L2 Cross-framework verified |
| Apple M2 Ultra | Ollama | IQ3_S | 116.9 tok/s | 13.19 GB | 1 | L2 Cross-framework verified |
| Apple M2 Ultra | Ollama | Q3_K_M | 114.2 tok/s | 13.52 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX Spark | llama.cpp | Q4_K_M | 50.1 tok/s | 15.88 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX Spark | llama.cpp | Q5_K_S | 49.3 tok/s | 17.22 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX Spark | Ollama | Q3_K_S | 56.8 tok/s | 13.02 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX Spark | Ollama | Q4_K_M | 51.3 tok/s | 15.88 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 4090 | llama.cpp | Q4_K_M | 147 tok/s | 10.64 GB | 2 | L2 Cross-framework verified |
| NVIDIA RTX 4090 | llama.cpp | Q4_K_S | 134.4 tok/s | 15.2 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 4090 | Strata | Q2_K | 254.9 tok/s | 11.34 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 3090 | Ollama | IQ3_S | 90.9 tok/s | 13.19 GB | 1 | L2 Cross-framework verified |
91 of these records are not linked to a specific quantized build; the full set is on the search page.
Community discussion
No discussions yet — start the first one.
Start a discussion