NVIDIA H20
GPU65
运行记录
23
覆盖模型
4
覆盖框架
L2 跨框架验证 × 65
证据分布
典型性能表现
同硬件上各模型 × 框架 × 量化的已上架实测均值。
- llama.cpp · IQ2_XS Decode 95.17 · Prefill 436.06 ·
- vLLM · Q3_K_M Decode 78.41 · Prefill 259.38 ·
- vLLM · Q5_K_S Decode 49.64 · Prefill 220.24 ·
- Strata · Q4_K_M Decode 73.54 · Prefill 286.94 ·
- llama.cpp · Q5_K_S Decode 116.43 · Prefill 457.51 ·
- Strata · IQ2_XS Decode 273.22 · Prefill 1069.33 ·
- llama.cpp · Q4_K_M Decode 61.75 · Prefill 319.38 ·
- Strata · Q4_K_M Decode 91.09 · Prefill 326.32 ·
- vLLM · Q4_K_M Decode 94.37 · Prefill 330.77 ·
- vLLM · Q6_K Decode 79.46 · Prefill 372.23 ·
- Ollama · Q3_K_S Decode 104.76 · Prefill 539.82 ·
- Ollama · IQ4_XS Decode 54.63 · Prefill 291.87 ·
- Ollama · Q3_K_M Decode 58.6 · Prefill 222.5 ·
- Strata · Q5_K_M Decode 68.61 · Prefill 223.86 ·
- llama.cpp · FP16 Decode 18.55 · Prefill 67.93 ·
- vLLM · IQ4_XS Decode 45.95 · Prefill 201.36 ·
- vLLM · FP16 Decode 25.76 · Prefill 119.08 ·
- vLLM · IQ2_XS Decode 68.56 · Prefill 374.49 ·
- Ollama · Q5_K_M Decode 36.62 · Prefill 154.85 ·
- llama.cpp · IQ2_XS Decode 42.1 · Prefill 229.32 ·
- Strata · Q2_K Decode 52.83 · Prefill 205.52 ·
- Strata · Q4_K_M Decode 39.14 · Prefill 148.55 ·
- vLLM · Q3_K_S Decode 48.48 · Prefill 199.47 ·
- Ollama · Q4_K_M Decode 32.97 · Prefill 115.23 ·
- vLLM · Q3_K_S Decode 118.48 · Prefill 535.48 ·
- vLLM · Q8_0 Decode 66.61 · Prefill 237.96 ·
- Ollama · Q3_K_M Decode 90.2 · Prefill 441.65 ·
- vLLM · Q3_K_S Decode 84.6 · Prefill 335.18 ·
- vLLM · Q4_K_M Decode 68.71 · Prefill 315.97 ·
- Strata · IQ4_XS Decode 97.77 · Prefill 362.6 ·
- vLLM · IQ2_XS Decode 63.17 · Prefill 260.61 ·
- vLLM · Q8_0 Decode 27.55 · Prefill 97.34 ·
- Strata · IQ3_S Decode 67.46 · Prefill 225.71 ·
- llama.cpp · Q3_K_L Decode 129.02 · Prefill 590.22 ·
- vLLM · Q4_K_M Decode 121.55 · Prefill 512.9 ·
- vLLM · Q8_0 Decode 82.63 · Prefill 440.06 ·
- vLLM · IQ3_S Decode 57.73 · Prefill 250.91 ·
- vLLM · Q3_K_L Decode 54.56 · Prefill 274.28 ·
- Strata · IQ3_S Decode 80.72 · Prefill 287.96 ·
- Strata · Q3_K_M Decode 74.4 · Prefill 349.95 ·
- llama.cpp · IQ3_S Decode 57.35 · Prefill 209.84 ·
- llama.cpp · Q5_K_S Decode 42.52 · Prefill 173.47 ·
- Strata · IQ3_S Decode 87.33 · Prefill 444.4 ·
- Strata · Q5_K_S Decode 59.66 · Prefill 317.37 ·
- vLLM · IQ3_XS Decode 39.95 · Prefill 179.27 ·
- vLLM · Q4_K_S Decode 32.57 · Prefill 141.17 ·
- Ollama · Q8_0 Decode 18.96 · Prefill 65.76 ·
- llama.cpp · IQ2_XS Decode 63.1 · Prefill 298.51 ·
- vLLM · Q3_K_L Decode 49.15 · Prefill 244.04 ·
- llama.cpp · Q3_K_M Decode 65.35 · Prefill 226.54 ·
Fit 判定
按显存与内存带宽做物理估算:选择模型,查看各量化能否装下及预计速度
估算公式:tok/s =(带宽 ÷ 模型占用)× 0.55,借鉴 llmfit;实测以运行记录为准
核心规格
- 厂商
- NVIDIA
- 显存
- 97 GB
- 显存带宽
- 4000 GB/s
运行记录
共 50 组配置
| 模型 | 框架 | 量化 | Decode | 生成耗时 | 样本数 |
|---|---|---|---|---|---|
| gemma-4-E4B-it | llama.cpp | IQ2_XS | 95.2 tok/s | — | 1 |
| gemma-4-E4B-it | vLLM | Q3_K_M | 78.4 tok/s | — | 1 |
| Ministral-3-8B-Instruct-2512 | vLLM | Q5_K_S | 49.6 tok/s | — | 1 |
| Ministral-3-8B-Instruct-2512 | Strata | Q4_K_M | 73.5 tok/s | — | 1 |
| iFlytek Spark-X2.5-4B | llama.cpp | Q5_K_S | 116.4 tok/s | — | 1 |
| iFlytek Spark-X2.5-4B | Strata | IQ2_XS | 273.2 tok/s | — | 1 |
| NVIDIA-Nemotron-3.5-Bolt-E4B-4GB | llama.cpp | Q4_K_M | 61.8 tok/s | — | 1 |
| NVIDIA-Nemotron-3.5-Bolt-E4B-4GB | Strata | Q4_K_M | 91.1 tok/s | — | 1 |
| Flint-40B-A2.5B | vLLM | Q4_K_M | 94.4 tok/s | — | 1 |
| Flint-40B-A2.5B | vLLM | Q6_K | 79.5 tok/s | — | 1 |
| Flint-40B-A2.5B | Ollama | Q3_K_S | 104.8 tok/s | — | 1 |
| Raon-1-mini-preview | Ollama | IQ4_XS | 54.6 tok/s | — | 1 |
| Raon-1-mini-preview | Ollama | Q3_K_M | 58.6 tok/s | — | 1 |
| Raon-1-mini-preview | Strata | Q5_K_M | 68.6 tok/s | — | 1 |
| mammoth-1 | llama.cpp | FP16 | 18.6 tok/s | — | 1 |
| mammoth-1 | vLLM | IQ4_XS | 46 tok/s | — | 1 |
| elephant-1 | vLLM | FP16 | 25.8 tok/s | — | 1 |
| elephant-1 | vLLM | IQ2_XS | 68.6 tok/s | — | 1 |
| elephant-1 | Ollama | Q5_K_M | 36.6 tok/s | — | 1 |
| Qwen3.8-27B | llama.cpp | IQ2_XS | 42.1 tok/s | — | 1 |
| Qwen3.8-27B | Strata | Q2_K | 52.8 tok/s | — | 1 |
| Qwen3.8-27B | Strata | Q4_K_M | 39.1 tok/s | — | 1 |
| Qwen3.6-27B | vLLM | Q3_K_S | 48.5 tok/s | — | 1 |
| Qwen3.6-27B | Ollama | Q4_K_M | 33 tok/s | — | 1 |
| Qwen3.6-35B-A3B | vLLM | Q3_K_S | 118.5 tok/s | — | 1 |
| Qwen3.6-35B-A3B | vLLM | Q8_0 | 66.6 tok/s | — | 1 |
| Qwen3.6-35B-A3B | Ollama | Q3_K_M | 90.2 tok/s | — | 1 |
| Qwen3.5-4B | vLLM | Q3_K_S | 84.6 tok/s | — | 1 |
| Qwen3.5-4B | vLLM | Q4_K_M | 68.7 tok/s | — | 1 |
| Qwen3.5-4B | Strata | IQ4_XS | 97.8 tok/s | — | 1 |
| Qwen3.5-27B | vLLM | IQ2_XS | 63.2 tok/s | — | 1 |
| Qwen3.5-27B | vLLM | Q8_0 | 27.6 tok/s | — | 1 |
| Qwen3.5-27B | Strata | IQ3_S | 67.5 tok/s | — | 1 |
| Qwen3.5-0.8B | llama.cpp | Q3_K_L | 129 tok/s | — | 1 |
| Qwen3.5-0.8B | vLLM | Q4_K_M | 121.5 tok/s | — | 1 |
| Qwen3.5-0.8B | vLLM | Q8_0 | 82.6 tok/s | — | 1 |
| Qwen3.8-9B | vLLM | IQ3_S | 57.7 tok/s | — | 1 |
| Qwen3.8-9B | vLLM | Q3_K_L | 54.6 tok/s | — | 1 |
| Qwen3.8-9B | Strata | IQ3_S | 80.7 tok/s | — | 1 |
| Qwen3.8-9B | Strata | Q3_K_M | 74.4 tok/s | — | 1 |
| Qwen3-8B | llama.cpp | IQ3_S | 57.4 tok/s | — | 1 |
| Qwen3-8B | llama.cpp | Q5_K_S | 42.5 tok/s | — | 1 |
| Qwen3-8B | Strata | IQ3_S | 87.3 tok/s | — | 1 |
| Qwen3-8B | Strata | Q5_K_S | 59.7 tok/s | — | 1 |
| gemma-4-31B-it | vLLM | IQ3_XS | 40 tok/s | — | 1 |
| gemma-4-31B-it | vLLM | Q4_K_S | 32.6 tok/s | — | 1 |
| gemma-4-31B-it | Ollama | Q8_0 | 19 tok/s | — | 1 |
| gemma-4-12B-it | llama.cpp | IQ2_XS | 63.1 tok/s | — | 1 |
| gemma-4-12B-it | vLLM | Q3_K_L | 49.1 tok/s | — | 1 |
| Llama-3.1-8B-Instruct | llama.cpp | Q3_K_M | 65.3 tok/s | — | 1 |