NVIDIA RTX 3090
GPUNVIDIA Ampere (GA102): 24 GB GDDR6X on a 384-bit bus, 936 GB/s of memory bandwidth, 350 W board power.
169
Run records
29
Models covered
7
Frameworks covered
L0 Self-reported × 70 · L2 Cross-framework verified × 99
Evidence mix
Typical measured performance
Published benchmark averages per model × framework × quantization on this hardware.
- llama.cpp · IQ3_S Decode 121.92 · Prefill 447.63 ·
- vLLM · IQ4_XS Decode 102.62 · Prefill 342.57 ·
- vLLM · Q4_K_S Decode 94.16 · Prefill 413.12 ·
- Strata · Q3_K_S Decode 133.25 · Prefill 628.12 ·
- llama.cpp · Q4_K_M Decode 94.58 · Prefill 353.95 ·
- Strata · Q4_K_M Decode 125.83 · Prefill 630.14 ·
- Ollama · Q4_K_S Decode 69.84 · Prefill 337.36 ·
- Strata · IQ1_S Decode 163.87 · Prefill 748 ·
- vLLM · IQ1_S Decode 96.12 · Prefill 373.79 ·
- llama.cpp · Q2_K Decode 137.84 · Prefill 685.72 ·
- llama.cpp · Q3_K_S Decode 115.05 · Prefill 556.74 ·
- Ollama · Q3_K_M Decode 101.46 · Prefill 355.86 ·
- Ollama · Q5_K_M Decode 80.28 · Prefill 423.81 ·
- Ollama · IQ1_S Decode 89.8 · Prefill 459.43 ·
- llama.cpp · Q5_K_S Decode 157.88 · Prefill 604.44 ·
- Ollama · IQ2_M Decode 246.2 · Prefill 1112.28 ·
- llama.cpp · IQ2_XS Decode 104.18 · Prefill 429.42 ·
- llama.cpp · Q8_0 Decode 44.9 · Prefill 150.43 ·
- Strata · IQ3_XS Decode 134.6 · Prefill 685.06 ·
- Strata · Q3_K_L Decode 103.74 · Prefill 393.26 ·
- vLLM · IQ2_XS Decode 113.39 · Prefill 440.93 ·
- Ollama · Q2_K Decode 98.87 · Prefill 381.75 ·
- llama.cpp · Q4_K_M Decode 79.26 · Prefill 473.41 ·
- Ollama · Q3_K_L Decode 64.09 · Prefill 302.97 ·
- vLLM · Q8_0 Decode 64.44 · Prefill 290.44 ·
- Ollama · IQ3_XS Decode 101.05 · Prefill 370.05 ·
- Ollama · Q5_K_S Decode 67.97 · Prefill 263.38 ·
- Strata · IQ3_S Decode 155.86 · Prefill 503.14 ·
- Strata · Q5_K_M Decode 101.92 · Prefill 340.28 ·
- llama.cpp · Q2_K Decode 287.44 · Prefill 1386.85 ·
- vLLM · FP16 Decode 113.95 · Prefill 381.37 ·
- vLLM · Q3_K_M Decode 250.47 · Prefill 1151.95 ·
- llama.cpp · IQ4_XS Decode 79.26 · Prefill 346.57 ·
- llama.cpp · Q3_K_M Decode 75.68 · Prefill 415.45 ·
- vLLM · Q3_K_M Decode 80.11 · Prefill 318.86 ·
- Ollama · IQ1_S Decode 76.57 · Prefill 351.59 ·
- Ollama · Q3_K_S Decode 109.99 · Prefill 558.1 ·
- Ollama · Q8_0 Decode 62.23 · Prefill 337.34 ·
- Strata · IQ4_XS Decode 157.12 · Prefill 630.97 ·
- vLLM · Q3_K_M Decode 217.39 · Prefill 922.41 ·
- vLLM · Q3_K_S Decode 241.45 · Prefill 1059.79 ·
- Ollama · Q3_K_L Decode 178.6 · Prefill 868.06 ·
- Ollama · Q5_K_M Decode 146.73 · Prefill 547.3 ·
- llama.cpp · Q3_K_L Decode 79.17 · Prefill 416.87 ·
- Ollama · Q5_K_S Decode 58.86 · Prefill 223.43 ·
- Ollama · Q8_0 Decode 46.15 · Prefill 154.63 ·
- vLLM · IQ3_XS Decode 108.64 · Prefill 552.19 ·
- Ollama · IQ4_XS Decode 96.65 · Prefill 382.18 ·
- Ollama · Q8_0 Decode 48.02 · Prefill 169.17 ·
- vLLM · Q6_K Decode 50.6 · Prefill 208.67 ·
Fit check
Physics-based estimate from VRAM and memory bandwidth: pick a model to see which quantizations fit and the expected speed
Formula: tok/s = (bandwidth ÷ model size) × 0.55, adapted from llmfit; see run records for measured numbers
Key specs
- Vendor
- NVIDIA
- Architecture
- Ampere
- VRAM
- 24 GB
- Memory bandwidth
- 936 GB/s
- TDP
- 350 W
- Release date
- 2020-09-24
Run records
50 configurations
| Model | Framework | Quantization | Decode | Gen time | Samples |
|---|---|---|---|---|---|
| gemma-4-E4B-it | llama.cpp | IQ3_S | 121.9 tok/s | — | 1 |
| gemma-4-E4B-it | vLLM | IQ4_XS | 102.6 tok/s | — | 1 |
| Ministral-3-8B-Instruct-2512 | vLLM | Q4_K_S | 94.2 tok/s | — | 1 |
| Ministral-3-8B-Instruct-2512 | Strata | Q3_K_S | 133.3 tok/s | — | 1 |
| NVIDIA-Nemotron-3.5-Bolt-E4B-4GB | llama.cpp | Q4_K_M | 94.6 tok/s | — | 1 |
| NVIDIA-Nemotron-3.5-Bolt-E4B-4GB | Strata | Q4_K_M | 125.8 tok/s | — | 1 |
| elephant-1 | Ollama | Q4_K_S | 69.8 tok/s | — | 1 |
| elephant-1 | Strata | IQ1_S | 163.9 tok/s | — | 1 |
| Qwen3.6-27B | vLLM | IQ1_S | 96.1 tok/s | — | 1 |
| Qwen3.5-4B | llama.cpp | Q2_K | 137.8 tok/s | — | 1 |
| Qwen3.5-4B | llama.cpp | Q3_K_S | 115 tok/s | — | 1 |
| Qwen3.5-4B | Ollama | Q3_K_M | 101.5 tok/s | — | 1 |
| Qwen3.5-4B | Ollama | Q5_K_M | 80.3 tok/s | — | 1 |
| Qwen3.5-27B | Ollama | IQ1_S | 89.8 tok/s | — | 1 |
| Qwen3.5-0.8B | llama.cpp | Q5_K_S | 157.9 tok/s | — | 1 |
| Qwen3.5-0.8B | Ollama | IQ2_M | 246.2 tok/s | — | 1 |
| Qwen3.8-9B | llama.cpp | IQ2_XS | 104.2 tok/s | — | 2 |
| Qwen3.8-9B | llama.cpp | Q8_0 | 44.9 tok/s | — | 1 |
| Qwen3.8-9B | Strata | IQ3_XS | 134.6 tok/s | — | 1 |
| Qwen3.8-9B | Strata | Q3_K_L | 103.7 tok/s | — | 1 |
| Qwen3-8B | vLLM | IQ2_XS | 113.4 tok/s | — | 1 |
| Qwen3-8B | Ollama | Q2_K | 98.9 tok/s | — | 1 |
| gemma-4-12B-it | llama.cpp | Q4_K_M | 79.3 tok/s | — | 2 |
| gemma-4-12B-it | Ollama | Q3_K_L | 64.1 tok/s | — | 1 |
| Llama-3.1-8B-Instruct | vLLM | Q8_0 | 64.4 tok/s | — | 1 |
| Llama-3.1-8B-Instruct | Ollama | IQ3_XS | 101 tok/s | — | 1 |
| Llama-3-8B-Instruct | Ollama | Q5_K_S | 68 tok/s | — | 1 |
| Llama-3-8B-Instruct | Strata | IQ3_S | 155.9 tok/s | — | 1 |
| Llama-3-8B-Instruct | Strata | Q5_K_M | 101.9 tok/s | — | 1 |
| Llama-3.2-3B-Instruct | llama.cpp | Q2_K | 287.4 tok/s | — | 2 |
| Llama-3.2-3B-Instruct | vLLM | FP16 | 114 tok/s | — | 1 |
| Llama-3.2-3B-Instruct | vLLM | Q3_K_M | 250.5 tok/s | — | 1 |
| DeepSeek-R1-Distill-Qwen-14B | llama.cpp | IQ4_XS | 79.3 tok/s | — | 1 |
| DeepSeek-R1-Distill-Qwen-14B | llama.cpp | Q3_K_M | 75.7 tok/s | — | 1 |
| DeepSeek-R1-Distill-Qwen-14B | vLLM | Q3_K_M | 80.1 tok/s | — | 1 |
| Qwen3.6-27B | Ollama | IQ1_S | 76.6 tok/s | — | 1 |
| Qwen3.5-4B | Ollama | Q3_K_S | 110 tok/s | — | 1 |
| Qwen3.5-4B | Ollama | Q8_0 | 62.2 tok/s | — | 1 |
| Qwen3.5-4B | Strata | IQ4_XS | 157.1 tok/s | — | 1 |
| Qwen3.5-0.8B | vLLM | Q3_K_M | 217.4 tok/s | — | 1 |
| Qwen3.5-0.8B | vLLM | Q3_K_S | 241.4 tok/s | — | 1 |
| Qwen3.5-0.8B | Ollama | Q3_K_L | 178.6 tok/s | — | 1 |
| Qwen3.5-0.8B | Ollama | Q5_K_M | 146.7 tok/s | — | 1 |
| Qwen3.8-9B | llama.cpp | Q3_K_L | 79.2 tok/s | — | 1 |
| Qwen3.8-9B | Ollama | Q5_K_S | 58.9 tok/s | — | 1 |
| Qwen3.8-9B | Ollama | Q8_0 | 46.1 tok/s | — | 1 |
| Qwen3-8B | vLLM | IQ3_XS | 108.6 tok/s | — | 1 |
| Qwen3-8B | Ollama | IQ4_XS | 96.7 tok/s | — | 2 |
| Qwen3-8B | Ollama | Q8_0 | 48 tok/s | — | 1 |
| gemma-4-12B-it | vLLM | Q6_K | 50.6 tok/s | — | 1 |