0
Quantized builds
75
Benchmark records
Quantized builds
Community and official quantized releases; "linked records" counts only benchmarks tied to that exact release.
No quantized builds cataloged yet.
Benchmark data
50 configurations
| Hardware | Framework | Quant | Decode | VRAM used | Samples | Evidence |
|---|---|---|---|---|---|---|
| NVIDIA RTX Spark | llama.cpp | FP16 | 4.7 tok/s | 110.2 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX Spark | llama.cpp | Q6_K | 6.4 tok/s | 67.2 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX Spark | llama.cpp | Q8_0 | 5.8 tok/s | 76.97 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX Spark | Ollama | FP16 | 4 tok/s | 110.2 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX Spark | Ollama | Q8_0 | 5.3 tok/s | 76.97 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 5090 | Strata | IQ2_M | 284.3 tok/s | 30.4 GB | 1 | L2 Cross-framework verified |
| AMD Ryzen AI Max+ 395 | llama.cpp | IQ2_M | 24.4 tok/s | 35.95 GB | 1 | L2 Cross-framework verified |
| AMD Ryzen AI Max+ 395 | llama.cpp | IQ2_XS | 25.1 tok/s | 33.35 GB | 1 | L2 Cross-framework verified |
| AMD Ryzen AI Max+ 395 | Ollama | IQ2_M | 24.9 tok/s | 35.95 GB | 1 | L2 Cross-framework verified |
| Apple M5 Max | llama.cpp | IQ1_S | 64.7 tok/s | 29.44 GB | 1 | L2 Cross-framework verified |
| Apple M5 Max | llama.cpp | IQ2_XS | 58.2 tok/s | 33.35 GB | 1 | L2 Cross-framework verified |
| Apple M5 Max | Ollama | Q4_K_M | 32.9 tok/s | 55.48 GB | 1 | L2 Cross-framework verified |
| Apple M5 Ultra | llama.cpp | Q3_K_L | 65.9 tok/s | 48.97 GB | 1 | L2 Cross-framework verified |
| Apple M5 Ultra | llama.cpp | Q8_0 | 38 tok/s | 76.97 GB | 1 | L2 Cross-framework verified |
| Apple M5 Ultra | Ollama | Q2_K | 75.9 tok/s | 37.91 GB | 1 | L2 Cross-framework verified |
| Apple M5 Ultra | Ollama | Q8_0 | 38.9 tok/s | 76.97 GB | 1 | L2 Cross-framework verified |
| NVIDIA H100 80GB HBM3 | vLLM | IQ3_XS | 338.7 tok/s | 42.46 GB | 1 | L2 Cross-framework verified |
| NVIDIA H100 80GB HBM3 | vLLM | Q3_K_M | 275.7 tok/s | 46.37 GB | 1 | L2 Cross-framework verified |
| NVIDIA H100 80GB HBM3 | Strata | Q5_K_S | 299.9 tok/s | 60.69 GB | 1 | L2 Cross-framework verified |
| NVIDIA H100 80GB HBM3 | Strata | Q6_K | 276.8 tok/s | 67.2 GB | 1 | L2 Cross-framework verified |
| NVIDIA H20 | vLLM | IQ3_XS | 40 tok/s | 42.46 GB | 1 | L2 Cross-framework verified |
| NVIDIA H20 | vLLM | Q4_K_S | 32.6 tok/s | 52.88 GB | 1 | L2 Cross-framework verified |
| NVIDIA H20 | Ollama | Q8_0 | 19 tok/s | 76.97 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX PRO 6000 Blackwell 96GB | vLLM | IQ2_XS | 189.8 tok/s | 33.35 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX PRO 6000 Blackwell 96GB | Ollama | IQ1_S | 184.4 tok/s | 29.44 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX PRO 6000 Blackwell 96GB | Ollama | Q4_K_S | 134.3 tok/s | 52.88 GB | 1 | L2 Cross-framework verified |
| Apple M2 Ultra | llama.cpp | Q4_K_M | 59 tok/s | 55.48 GB | 1 | L2 Cross-framework verified |
| Apple M2 Ultra | Ollama | IQ2_XS | 73.2 tok/s | 33.35 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX Spark | llama.cpp | Q4_K_M | 8.8 tok/s | 55.48 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX Spark | Ollama | Q3_K_S | 10.2 tok/s | 44.41 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX Spark | Ollama | Q4_K_M | 8 tok/s | 55.48 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 5090 | vLLM | IQ2_M | 219.7 tok/s | 30.4 GB | 1 | L2 Cross-framework verified |
| AMD Ryzen AI Max+ 395 | Strata | IQ3_XS | 81.3 tok/s | 42.46 GB | 1 | L2 Cross-framework verified |
| AMD Ryzen AI Max+ 395 | Strata | Q3_K_S | 82.3 tok/s | 44.41 GB | 1 | L2 Cross-framework verified |
| Apple M4 Pro | vLLM | IQ2_M | 64.9 tok/s | 35.95 GB | 1 | L2 Cross-framework verified |
| Apple M5 Pro | Ollama | Q3_K_M | 62.3 tok/s | 46.37 GB | 1 | L2 Cross-framework verified |
| Apple M5 Pro | Strata | IQ3_XS | 111.4 tok/s | 42.46 GB | 1 | L2 Cross-framework verified |
| Apple M5 Max | Strata | IQ4_XS | 138.4 tok/s | 50.27 GB | 1 | L2 Cross-framework verified |
| Apple M5 Max | Strata | Q3_K_M | 149.4 tok/s | 46.37 GB | 1 | L2 Cross-framework verified |
| NVIDIA DGX Spark | llama.cpp | IQ3_XS | 91.6 tok/s | 42.46 GB | 1 | L2 Cross-framework verified |
| NVIDIA DGX Spark | Ollama | Q6_K | 50.1 tok/s | 67.2 GB | 1 | L2 Cross-framework verified |
| NVIDIA Jetson AGX Orin 64GB | Ollama | IQ3_XS | 29.4 tok/s | 42.46 GB | 1 | L2 Cross-framework verified |
| NVIDIA Jetson AGX Orin 64GB | Ollama | Q3_K_M | 26.6 tok/s | 46.37 GB | 1 | L2 Cross-framework verified |
| Apple M3 Max | llama.cpp | Q2_K | 64.5 tok/s | 34.2 GB | 1 | L2 Cross-framework verified |
| Apple M3 Max | Ollama | Q2_K | 65.7 tok/s | 34.2 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 5090 | llama.cpp | IQ2_M | 159.3 tok/s | 30.4 GB | 1 | L2 Cross-framework verified |
| NVIDIA RTX 5090 | Ollama | IQ1_S | 146.2 tok/s | 29.44 GB | 1 | L2 Cross-framework verified |
| Apple M4 Max | llama.cpp | Q3_K_M | 93 tok/s | 46.37 GB | 1 | L2 Cross-framework verified |
| Apple M4 Max | llama.cpp | Q8_0 | 53.1 tok/s | 76.97 GB | 1 | L2 Cross-framework verified |
| Apple M4 Max | vLLM | Q2_K | 111.5 tok/s | 37.91 GB | 1 | L2 Cross-framework verified |
75 of these records are not linked to a specific quantized build; the full set is on the search page.
Community discussion
No discussions yet — start the first one.
Start a discussion