0
Quantized builds
12
Benchmark records
Quantized builds
Community and official quantized releases; "linked records" counts only benchmarks tied to that exact release.
No quantized builds cataloged yet.
Benchmark data
12 configurations
| Hardware | Framework | Quant | Decode | VRAM used | Samples | Evidence |
|---|---|---|---|---|---|---|
| NVIDIA RTX 5070 Ti | vLLM | IQ2_XS | 75.4 tok/s | 10.6 GB | 1 | L0 Self-reported |
| NVIDIA RTX 5070 Ti | vLLM | Q4_K_M | 41.8 tok/s | 16 GB | 1 | L0 Self-reported |
| NVIDIA RTX 5070 Ti | llama.cpp | IQ2_XS | 69.2 tok/s | 10.6 GB | 1 | L0 Self-reported |
| NVIDIA RTX 5070 Ti | llama.cpp | Q4_K_M | 38.4 tok/s | 16 GB | 1 | L0 Self-reported |
| NVIDIA RTX 5090 | vLLM | IQ2_XS | 150.8 tok/s | 10.6 GB | 1 | L0 Self-reported |
| NVIDIA RTX 5090 | vLLM | Q4_K_M | 83.7 tok/s | 18.7 GB | 1 | L0 Self-reported |
| NVIDIA RTX 5090 | llama.cpp | IQ2_XS | 138.4 tok/s | 10.6 GB | 1 | L0 Self-reported |
| NVIDIA RTX 5090 | llama.cpp | Q4_K_M | 76.8 tok/s | 18.7 GB | 1 | L0 Self-reported |
| NVIDIA RTX 4090 | vLLM | IQ2_XS | 84.8 tok/s | 10.6 GB | 1 | L0 Self-reported |
| NVIDIA RTX 4090 | vLLM | Q4_K_M | 47.1 tok/s | 18.7 GB | 1 | L0 Self-reported |
| NVIDIA RTX 4090 | llama.cpp | IQ2_XS | 77.8 tok/s | 10.6 GB | 1 | L0 Self-reported |
| NVIDIA RTX 4090 | llama.cpp | Q4_K_M | 43.2 tok/s | 18.7 GB | 1 | L0 Self-reported |
12 of these records are not linked to a specific quantized build; the full set is on the search page.
Community discussion
No discussions yet — start the first one.
Start a discussion