NVIDIA RTX PRO 6000 Blackwell 96GB
GPU74
运行记录
23
覆盖模型
4
覆盖框架
L0 自报 × 6 · L2 跨框架验证 × 68
证据分布
典型性能表现
同硬件上各模型 × 框架 × 量化的已上架实测均值。
- vLLM · IQ3_XS Decode 344.94 · Prefill 1590.34 ·
- Ollama · IQ2_XS Decode 387.68 · Prefill 2077.1 ·
- vLLM · Q2_K Decode 326.52 · Prefill 1378.65 ·
- Strata · Q5_K_M Decode 252.2 · Prefill 1116.29 ·
- llama.cpp · Q6_K Decode 350.22 · Prefill 1535.88 ·
- Strata · Q3_K_S Decode 316.72 · Prefill 1709.01 ·
- vLLM · BF16 Decode 177.62 · Prefill 884.45 ·
- Ollama · Q4_K_M Decode 264.78 · Prefill 915.82 ·
- vLLM · Q3_K_L Decode 327.3 · Prefill 1746.79 ·
- Strata · Q3_K_S Decode 343.48 · Prefill 1507.36 ·
- vLLM · IQ3_S Decode 304.11 · Prefill 1069.77 ·
- vLLM · IQ4_XS Decode 310.08 · Prefill 1540.64 ·
- Ollama · Q4_K_M Decode 227.94 · Prefill 779.84 ·
- vLLM · IQ3_XS Decode 228.17 · Prefill 758.2 ·
- vLLM · Q5_K_M Decode 145.66 · Prefill 731.49 ·
- Ollama · IQ3_XS Decode 214.08 · Prefill 806.7 ·
- Ollama · Q5_K_S Decode 135.94 · Prefill 501.24 ·
- Ollama · IQ1_S Decode 275.81 · Prefill 1195.7 ·
- Ollama · IQ4_XS Decode 194.74 · Prefill 889.56 ·
- Strata · IQ3_XS Decode 354.49 · Prefill 1856.14 ·
- Strata · Q4_K_S Decode 293.11 · Prefill 1298.77 ·
- llama.cpp · Q5_K_S Decode 112.25 · Prefill 455.96 ·
- Ollama · IQ1_S Decode 182.59 · Prefill 968.64 ·
- Ollama · Q6_K Decode 86.06 · Prefill 324.27 ·
- llama.cpp · IQ1_S Decode 267.34 · Prefill 934.02 ·
- llama.cpp · Q8_0 Decode 105.75 · Prefill 445.38 ·
- vLLM · IQ3_S Decode 220.08 · Prefill 989.75 ·
- vLLM · IQ3_XS Decode 319.33 · Prefill 1553.22 ·
- Ollama · Q5_K_S Decode 301.35 · Prefill 1440.97 ·
- vLLM · Q3_K_M Decode 336.72 · Prefill 1847.35 ·
- vLLM · Q6_K Decode 245.04 · Prefill 1242.69 ·
- Ollama · Q3_K_L Decode 279.88 · Prefill 1395.76 ·
- Ollama · Q5_K_S Decode 216.95 · Prefill 820.67 ·
- llama.cpp · Q4_K_M Decode 181.73 · Prefill 687.5 ·
- llama.cpp · Q6_K Decode 133.14 · Prefill 542.78 ·
- Ollama · FP16 Decode 79.58 · Prefill 274.78 ·
- Ollama · Q3_K_L Decode 177.2 · Prefill 930.23 ·
- vLLM · FP16 Decode 273.07 · Prefill 1476.93 ·
- vLLM · IQ3_S Decode 331.56 · Prefill 1317.38 ·
- Strata · Q4_K_S Decode 356.54 · Prefill 1576.84 ·
- vLLM · Q4_K_S Decode 212.78 · Prefill 1065.77 ·
- Ollama · Q3_K_M Decode 221.75 · Prefill 855.6 ·
- Ollama · Q5_K_M Decode 148.52 · Prefill 762.34 ·
- llama.cpp · Q2_K Decode 305.21 · Prefill 1181.07 ·
- Strata · IQ3_S Decode 374.23 · Prefill 1951.46 ·
- Strata · Q3_K_L Decode 317.22 · Prefill 1349.4 ·
- vLLM · IQ2_XS Decode 189.85 · Prefill 737.75 ·
- Ollama · IQ1_S Decode 184.42 · Prefill 842.53 ·
- Ollama · Q4_K_S Decode 134.3 · Prefill 614.37 ·
- Ollama · IQ3_XS Decode 204.74 · Prefill 1066.91 ·
Fit 判定
按显存与内存带宽做物理估算:选择模型,查看各量化能否装下及预计速度
估算公式:tok/s =(带宽 ÷ 模型占用)× 0.55,借鉴 llmfit;实测以运行记录为准
核心规格
- 厂商
- NVIDIA
- 显存
- 96 GB
- 显存带宽
- 1792 GB/s
运行记录
共 50 组配置
| 模型 | 框架 | 量化 | Decode | 生成耗时 | 样本数 |
|---|---|---|---|---|---|
| gemma-4-E4B-it | vLLM | IQ3_XS | 344.9 tok/s | — | 1 |
| gemma-4-E4B-it | Ollama | IQ2_XS | 387.7 tok/s | — | 1 |
| Ministral-3-8B-Instruct-2512 | vLLM | Q2_K | 326.5 tok/s | — | 1 |
| Ministral-3-8B-Instruct-2512 | Strata | Q5_K_M | 252.2 tok/s | — | 1 |
| iFlytek Spark-X2.5-4B | llama.cpp | Q6_K | 350.2 tok/s | — | 1 |
| iFlytek Spark-X2.5-4B | Strata | Q3_K_S | 316.7 tok/s | — | 1 |
| NVIDIA-Nemotron-3.5-Bolt-E4B-4GB | vLLM | BF16 | 177.6 tok/s | — | 1 |
| NVIDIA-Nemotron-3.5-Bolt-E4B-4GB | Ollama | Q4_K_M | 264.8 tok/s | — | 1 |
| Flint-40B-A2.5B | vLLM | Q3_K_L | 327.3 tok/s | — | 1 |
| Flint-40B-A2.5B | Strata | Q3_K_S | 343.5 tok/s | — | 1 |
| Raon-1-mini-preview | vLLM | IQ3_S | 304.1 tok/s | — | 1 |
| Raon-1-mini-preview | vLLM | IQ4_XS | 310.1 tok/s | — | 1 |
| Raon-1-mini-preview | Ollama | Q4_K_M | 227.9 tok/s | — | 1 |
| mammoth-1 | vLLM | IQ3_XS | 228.2 tok/s | — | 1 |
| mammoth-1 | vLLM | Q5_K_M | 145.7 tok/s | — | 1 |
| mammoth-1 | Ollama | IQ3_XS | 214.1 tok/s | — | 1 |
| mammoth-1 | Ollama | Q5_K_S | 135.9 tok/s | — | 1 |
| elephant-1 | Ollama | IQ1_S | 275.8 tok/s | — | 1 |
| elephant-1 | Ollama | IQ4_XS | 194.7 tok/s | — | 1 |
| elephant-1 | Strata | IQ3_XS | 354.5 tok/s | — | 1 |
| elephant-1 | Strata | Q4_K_S | 293.1 tok/s | — | 1 |
| Qwen3.8-27B | llama.cpp | Q5_K_S | 112.3 tok/s | — | 1 |
| Qwen3.8-27B | Ollama | IQ1_S | 182.6 tok/s | — | 1 |
| Qwen3.8-27B | Ollama | Q6_K | 86.1 tok/s | — | 1 |
| Qwen3.6-27B | llama.cpp | IQ1_S | 267.3 tok/s | — | 1 |
| Qwen3.6-27B | llama.cpp | Q8_0 | 105.8 tok/s | — | 1 |
| Qwen3.6-27B | vLLM | IQ3_S | 220.1 tok/s | — | 1 |
| Qwen3.6-35B-A3B | vLLM | IQ3_XS | 319.3 tok/s | — | 1 |
| Qwen3.6-35B-A3B | Ollama | Q5_K_S | 301.4 tok/s | — | 1 |
| Qwen3.5-4B | vLLM | Q3_K_M | 336.7 tok/s | — | 1 |
| Qwen3.5-4B | vLLM | Q6_K | 245 tok/s | — | 1 |
| Qwen3.5-4B | Ollama | Q3_K_L | 279.9 tok/s | — | 1 |
| Qwen3.5-4B | Ollama | Q5_K_S | 216.9 tok/s | — | 1 |
| Qwen3.5-27B | llama.cpp | Q4_K_M | 181.7 tok/s | — | 1 |
| Qwen3.5-27B | llama.cpp | Q6_K | 133.1 tok/s | — | 1 |
| Qwen3.5-27B | Ollama | FP16 | 79.6 tok/s | — | 1 |
| Qwen3.5-27B | Ollama | Q3_K_L | 177.2 tok/s | — | 1 |
| Qwen3.5-0.8B | vLLM | FP16 | 273.1 tok/s | — | 1 |
| Qwen3.5-0.8B | vLLM | IQ3_S | 331.6 tok/s | — | 1 |
| Qwen3.5-0.8B | Strata | Q4_K_S | 356.5 tok/s | — | 1 |
| Qwen3.8-9B | vLLM | Q4_K_S | 212.8 tok/s | — | 1 |
| Qwen3.8-9B | Ollama | Q3_K_M | 221.8 tok/s | — | 1 |
| Qwen3.8-9B | Ollama | Q5_K_M | 148.5 tok/s | — | 1 |
| Qwen3-8B | llama.cpp | Q2_K | 305.2 tok/s | — | 1 |
| Qwen3-8B | Strata | IQ3_S | 374.2 tok/s | — | 1 |
| Qwen3-8B | Strata | Q3_K_L | 317.2 tok/s | — | 1 |
| gemma-4-31B-it | vLLM | IQ2_XS | 189.8 tok/s | — | 1 |
| gemma-4-31B-it | Ollama | IQ1_S | 184.4 tok/s | — | 1 |
| gemma-4-31B-it | Ollama | Q4_K_S | 134.3 tok/s | — | 1 |
| gemma-4-12B-it | Ollama | IQ3_XS | 204.7 tok/s | — | 1 |