NVIDIA RTX 3060 12GB
GPUNVIDIA Ampere (GA106): 12 GB GDDR6 on a 192-bit bus, 360 GB/s of memory bandwidth, 170 W board power. The 12 GB SKU, distinct from the 8 GB variant.
83
Run records
21
Models covered
4
Frameworks covered
L0 Self-reported × 13 · L1 Reproduced × 1 · L2 Cross-framework verified × 69
Evidence mix
Typical measured performance
Published benchmark averages per model × framework × quantization on this hardware.
- vLLM · IQ3_XS Decode 89.05 · Prefill 461.73 ·
- Ollama · Q3_K_S Decode 79.42 · Prefill 278.2 ·
- Ollama · IQ2_M Decode 182.69 · Prefill 954.31 ·
- Strata · Q4_K_S Decode 203.46 · Prefill 1004.82 ·
- vLLM · Q2_K Decode 70.07 · Prefill 259.24 ·
- Ollama · IQ1_S Decode 110.04 · Prefill 354.52 ·
- Strata · Q3_K_M Decode 112.29 · Prefill 547.52 ·
- llama.cpp · FP16 Decode 72.92 · Prefill 274.88 ·
- llama.cpp · Q5_K_S Decode 132.45 · Prefill 600.24 ·
- Ollama · Q3_K_S Decode 138.92 · Prefill 607.45 ·
- Ollama · Q2_K Decode 64.94 · Prefill 300.22 ·
- vLLM · Q3_K_S Decode 73.32 · Prefill 238.24 ·
- Ollama · IQ3_S Decode 58.42 · Prefill 197.59 ·
- Strata · IQ1_S Decode 99.13 · Prefill 497.2 ·
- llama.cpp · IQ3_S Decode 72.47 · Prefill 300.81 ·
- vLLM · IQ2_XS Decode 94.63 · Prefill 430.36 ·
- Strata · Q3_K_M Decode 95.19 · Prefill 317.89 ·
- vLLM · Q4_K_S Decode 182.86 · Prefill 651.97 ·
- vLLM · Q5_K_S Decode 145.02 · Prefill 478.55 ·
- Strata · Q6_K Decode 168.05 · Prefill 844.89 ·
- Ollama · IQ2_M Decode 91.19 · Prefill 447.34 ·
- Ollama · Q4_K_M Decode 66.59 · Prefill 338.55 ·
- Ollama · FP16 Decode 65.06 · Prefill 244.73 ·
- Ollama · Q2_K Decode 166.72 · Prefill 642.75 ·
- Strata · FP16 Decode 103.42 · Prefill 466.54 ·
- vLLM · IQ3_XS Decode 64.02 · Prefill 298.14 ·
- llama.cpp · IQ2_M Decode 70.85 · Prefill 298.67 ·
- vLLM · IQ3_S Decode 64 · Prefill 232.21 ·
- vLLM · IQ2_M Decode 92.78 · Prefill 311.99 ·
- Strata · IQ1_S Decode 126.53 · Prefill 600.27 ·
- Strata · Q3_K_S Decode 98.81 · Prefill 492.63 ·
- llama.cpp · IQ3_S Decode 193.44 · Prefill 733.2 ·
- llama.cpp · Q5_K_S Decode 126.84 · Prefill 630.8 ·
- vLLM · Q5_K_M Decode 143.38 · Prefill 521.02 ·
- llama.cpp · Q3_K_L Decode 55.93 · Prefill 219.18 ·
- llama.cpp · Q3_K_M Decode 58.85 · Prefill 193.51 ·
- vLLM · IQ4_XS Decode 53.14 · Prefill 228.98 ·
- llama.cpp · IQ3_S Decode 105.18 · Prefill 415.05 ·
- Ollama · IQ3_S Decode 95.65 · Prefill 519.99 ·
- Ollama · Q3_K_M Decode 97.19 · Prefill 407.59 ·
- Strata · Q5_K_S Decode 113.91 · Prefill 618.95 ·
- Strata · Q6_K Decode 100.84 · Prefill 413.9 ·
- llama.cpp · Q3_K_S Decode 187.82 · Prefill 792.93 ·
- Ollama · Q3_K_L Decode 167.74 · Prefill 790.34 ·
- Ollama · Q5_K_M Decode 118.62 · Prefill 422.11 ·
- Strata · Q3_K_M Decode 279.25 · Prefill 1294.74 ·
- Strata · Q5_K_S Decode 214.78 · Prefill 1075.84 ·
- Strata · Q2_K Decode 114.46 · Prefill 600.51 ·
- llama.cpp · IQ3_S Decode 81.43 · Prefill 366.11 ·
- llama.cpp · Q2_K Decode 169.7 · Prefill 583.54 ·
Fit check
Physics-based estimate from VRAM and memory bandwidth: pick a model to see which quantizations fit and the expected speed
Formula: tok/s = (bandwidth ÷ model size) × 0.55, adapted from llmfit; see run records for measured numbers
Key specs
- Vendor
- NVIDIA
- Architecture
- Ampere
- VRAM
- 12 GB
- Memory bandwidth
- 360 GB/s
- TDP
- 170 W
- Release date
- 2021-02-25
Run records
50 configurations
| Model | Framework | Quantization | Decode | Gen time | Samples |
|---|---|---|---|---|---|
| gemma-4-E4B-it | vLLM | IQ3_XS | 89 tok/s | — | 1 |
| gemma-4-E4B-it | Ollama | Q3_K_S | 79.4 tok/s | — | 1 |
| iFlytek Spark-X2.5-4B | Ollama | IQ2_M | 182.7 tok/s | — | 1 |
| iFlytek Spark-X2.5-4B | Strata | Q4_K_S | 203.5 tok/s | — | 1 |
| elephant-1 | vLLM | Q2_K | 70.1 tok/s | — | 1 |
| Qwen3.5-4B | Ollama | IQ1_S | 110 tok/s | — | 1 |
| Qwen3.5-4B | Strata | Q3_K_M | 112.3 tok/s | — | 1 |
| Qwen3.5-0.8B | llama.cpp | FP16 | 72.9 tok/s | — | 1 |
| Qwen3.5-0.8B | llama.cpp | Q5_K_S | 132.4 tok/s | — | 2 |
| Qwen3.5-0.8B | Ollama | Q3_K_S | 138.9 tok/s | — | 1 |
| Qwen3.8-9B | Ollama | Q2_K | 64.9 tok/s | — | 1 |
| Qwen3-8B | vLLM | Q3_K_S | 73.3 tok/s | — | 1 |
| Qwen3-8B | Ollama | IQ3_S | 58.4 tok/s | — | 1 |
| gemma-4-12B-it | Strata | IQ1_S | 99.1 tok/s | — | 1 |
| Llama-3.1-8B-Instruct | llama.cpp | IQ3_S | 72.5 tok/s | — | 1 |
| Llama-3.1-8B-Instruct | vLLM | IQ2_XS | 94.6 tok/s | — | 1 |
| Llama-3-8B-Instruct | Strata | Q3_K_M | 95.2 tok/s | — | 1 |
| Llama-3.2-3B-Instruct | vLLM | Q4_K_S | 182.9 tok/s | — | 1 |
| Llama-3.2-3B-Instruct | vLLM | Q5_K_S | 145 tok/s | — | 1 |
| Llama-3.2-3B-Instruct | Strata | Q6_K | 168.1 tok/s | — | 1 |
| Qwen3.5-4B | Ollama | IQ2_M | 91.2 tok/s | — | 1 |
| Qwen3.5-4B | Ollama | Q4_K_M | 66.6 tok/s | — | 1 |
| Qwen3.5-0.8B | Ollama | FP16 | 65.1 tok/s | — | 1 |
| Qwen3.5-0.8B | Ollama | Q2_K | 166.7 tok/s | — | 1 |
| Qwen3.5-0.8B | Strata | FP16 | 103.4 tok/s | — | 1 |
| Qwen3.8-9B | vLLM | IQ3_XS | 64 tok/s | — | 1 |
| Qwen3-8B | llama.cpp | IQ2_M | 70.8 tok/s | — | 1 |
| Qwen3-8B | vLLM | IQ3_S | 64 tok/s | — | 1 |
| Llama-3.1-8B-Instruct | vLLM | IQ2_M | 92.8 tok/s | — | 1 |
| Llama-3-8B-Instruct | Strata | IQ1_S | 126.5 tok/s | — | 1 |
| Llama-3-8B-Instruct | Strata | Q3_K_S | 98.8 tok/s | — | 1 |
| Llama-3.2-3B-Instruct | llama.cpp | IQ3_S | 193.4 tok/s | — | 1 |
| Llama-3.2-3B-Instruct | llama.cpp | Q5_K_S | 126.8 tok/s | — | 1 |
| Llama-3.2-3B-Instruct | vLLM | Q5_K_M | 143.4 tok/s | — | 1 |
| Kimi-K3 | llama.cpp | Q3_K_L | 55.9 tok/s | — | 1 |
| Kimi-K3 | llama.cpp | Q3_K_M | 58.9 tok/s | — | 1 |
| Kimi-K3 | vLLM | IQ4_XS | 53.1 tok/s | — | 1 |
| Qwen3.5-4B | llama.cpp | IQ3_S | 105.2 tok/s | — | 1 |
| Qwen3.5-4B | Ollama | IQ3_S | 95.7 tok/s | — | 1 |
| Qwen3.5-4B | Ollama | Q3_K_M | 97.2 tok/s | — | 1 |
| Qwen3.5-4B | Strata | Q5_K_S | 113.9 tok/s | — | 1 |
| Qwen3.5-4B | Strata | Q6_K | 100.8 tok/s | — | 1 |
| Qwen3.5-0.8B | llama.cpp | Q3_K_S | 187.8 tok/s | — | 1 |
| Qwen3.5-0.8B | Ollama | Q3_K_L | 167.7 tok/s | — | 1 |
| Qwen3.5-0.8B | Ollama | Q5_K_M | 118.6 tok/s | — | 1 |
| Qwen3.5-0.8B | Strata | Q3_K_M | 279.3 tok/s | — | 1 |
| Qwen3.5-0.8B | Strata | Q5_K_S | 214.8 tok/s | — | 1 |
| Qwen3.8-9B | Strata | Q2_K | 114.5 tok/s | — | 1 |
| Qwen3-8B | llama.cpp | IQ3_S | 81.4 tok/s | — | 1 |
| Qwen3-1.7B | llama.cpp | Q2_K | 169.7 tok/s | — | 1 |