0
Quantized builds
42
Benchmark records
Quantized builds
Community and official quantized releases; "linked records" counts only benchmarks tied to that exact release.
No quantized builds cataloged yet.
Benchmark data
42 configurations
| Hardware | Framework | Quant | Decode | VRAM used | Samples | Evidence |
|---|---|---|---|---|---|---|
| NVIDIA RTX 5060 SUPER | vLLM | Q8_0 | 29.3 tok/s | 12 GB | 1 | L0 Self-reported |
| NVIDIA RTX 5060 SUPER | vLLM | Q5_K_M | 42.8 tok/s | 9.4 GB | 1 | L0 Self-reported |
| NVIDIA RTX 5060 SUPER | vLLM | Q4_K_M | 52.2 tok/s | 7.8 GB | 1 | L0 Self-reported |
| NVIDIA RTX 5060 SUPER | llama.cpp | Q8_0 | 26.9 tok/s | 12 GB | 1 | L0 Self-reported |
| NVIDIA RTX 5060 SUPER | llama.cpp | Q5_K_M | 39.3 tok/s | 9.4 GB | 1 | L0 Self-reported |
| NVIDIA RTX 5060 SUPER | llama.cpp | Q4_K_M | 47.9 tok/s | 7.8 GB | 1 | L0 Self-reported |
| NVIDIA RTX 5070 SUPER | vLLM | Q8_0 | 58.6 tok/s | 13.5 GB | 1 | L0 Self-reported |
| NVIDIA RTX 5070 SUPER | vLLM | Q5_K_M | 85.6 tok/s | 9.4 GB | 1 | L0 Self-reported |
| NVIDIA RTX 5070 SUPER | vLLM | Q4_K_M | 104.3 tok/s | 7.8 GB | 1 | L0 Self-reported |
| NVIDIA RTX 5070 SUPER | llama.cpp | Q8_0 | 53.8 tok/s | 13.5 GB | 1 | L0 Self-reported |
| NVIDIA RTX 5070 SUPER | llama.cpp | Q5_K_M | 78.5 tok/s | 9.4 GB | 1 | L0 Self-reported |
| NVIDIA RTX 5070 SUPER | llama.cpp | Q4_K_M | 95.7 tok/s | 7.8 GB | 1 | L0 Self-reported |
| NVIDIA RTX 3090 | vLLM | Q8_0 | 61.2 tok/s | 13.5 GB | 1 | L0 Self-reported |
| NVIDIA RTX 3090 | vLLM | Q5_K_M | 89.4 tok/s | 9.4 GB | 1 | L0 Self-reported |
| NVIDIA RTX 3090 | vLLM | Q4_K_M | 109 tok/s | 7.8 GB | 1 | L0 Self-reported |
| NVIDIA RTX 3090 | llama.cpp | Q8_0 | 56.2 tok/s | 13.5 GB | 1 | L0 Self-reported |
| NVIDIA RTX 3090 | llama.cpp | Q5_K_M | 82 tok/s | 9.4 GB | 1 | L0 Self-reported |
| NVIDIA RTX 3090 | llama.cpp | Q4_K_M | 100 tok/s | 7.8 GB | 1 | L0 Self-reported |
| NVIDIA RTX 5070 Ti | vLLM | Q8_0 | 58.6 tok/s | 13.5 GB | 1 | L0 Self-reported |
| NVIDIA RTX 5070 Ti | vLLM | Q5_K_M | 85.6 tok/s | 9.4 GB | 1 | L0 Self-reported |
| NVIDIA RTX 5070 Ti | vLLM | Q4_K_M | 104.3 tok/s | 7.8 GB | 1 | L0 Self-reported |
| NVIDIA RTX 5070 Ti | llama.cpp | Q8_0 | 53.8 tok/s | 13.5 GB | 1 | L0 Self-reported |
| NVIDIA RTX 5070 Ti | llama.cpp | Q5_K_M | 78.5 tok/s | 9.4 GB | 1 | L0 Self-reported |
| NVIDIA RTX 5070 Ti | llama.cpp | Q4_K_M | 95.7 tok/s | 7.8 GB | 1 | L0 Self-reported |
| NVIDIA RTX 5070 | vLLM | Q8_0 | 43.9 tok/s | 12 GB | 1 | L0 Self-reported |
| NVIDIA RTX 5070 | vLLM | Q5_K_M | 64.2 tok/s | 9.4 GB | 1 | L0 Self-reported |
| NVIDIA RTX 5070 | vLLM | Q4_K_M | 78.2 tok/s | 7.8 GB | 1 | L0 Self-reported |
| NVIDIA RTX 5070 | llama.cpp | Q8_0 | 40.3 tok/s | 12 GB | 1 | L0 Self-reported |
| NVIDIA RTX 5070 | llama.cpp | Q5_K_M | 58.9 tok/s | 9.4 GB | 1 | L0 Self-reported |
| NVIDIA RTX 5070 | llama.cpp | Q4_K_M | 71.8 tok/s | 7.8 GB | 1 | L0 Self-reported |
| NVIDIA RTX 5090 | vLLM | Q8_0 | 117.2 tok/s | 13.5 GB | 1 | L0 Self-reported |
| NVIDIA RTX 5090 | vLLM | Q5_K_M | 171.1 tok/s | 9.4 GB | 1 | L0 Self-reported |
| NVIDIA RTX 5090 | vLLM | Q4_K_M | 208.7 tok/s | 7.8 GB | 1 | L0 Self-reported |
| NVIDIA RTX 5090 | llama.cpp | Q8_0 | 107.5 tok/s | 13.5 GB | 1 | L0 Self-reported |
| NVIDIA RTX 5090 | llama.cpp | Q5_K_M | 157.1 tok/s | 9.4 GB | 1 | L0 Self-reported |
| NVIDIA RTX 5090 | llama.cpp | Q4_K_M | 191.5 tok/s | 7.8 GB | 1 | L0 Self-reported |
| NVIDIA RTX 4090 | vLLM | Q8_0 | 65.9 tok/s | 13.5 GB | 1 | L0 Self-reported |
| NVIDIA RTX 4090 | vLLM | Q5_K_M | 96.3 tok/s | 9.4 GB | 1 | L0 Self-reported |
| NVIDIA RTX 4090 | vLLM | Q4_K_M | 117.4 tok/s | 7.8 GB | 1 | L0 Self-reported |
| NVIDIA RTX 4090 | llama.cpp | Q8_0 | 60.5 tok/s | 13.5 GB | 1 | L0 Self-reported |
| NVIDIA RTX 4090 | llama.cpp | Q5_K_M | 88.3 tok/s | 9.4 GB | 1 | L0 Self-reported |
| NVIDIA RTX 4090 | llama.cpp | Q4_K_M | 107.7 tok/s | 7.8 GB | 1 | L0 Self-reported |
42 of these records are not linked to a specific quantized build; the full set is on the search page.
Community discussion
No discussions yet — start the first one.
Start a discussion