NVIDIA H100 80GB HBM3
GPU60
运行记录
22
覆盖模型
4
覆盖框架
L2 跨框架验证 × 60
证据分布
典型性能表现
同硬件上各模型 × 框架 × 量化的已上架实测均值。
- llama.cpp · IQ3_XS Decode 297.43 · Prefill 1240.57 ·
- vLLM · Q3_K_S Decode 343.31 · Prefill 1851.05 ·
- llama.cpp · Q8_0 Decode 271.67 · Prefill 1471.82 ·
- vLLM · Q3_K_M Decode 321.73 · Prefill 1464.84 ·
- Ollama · IQ2_XS Decode 285.58 · Prefill 1151.82 ·
- Strata · Q3_K_S Decode 291.71 · Prefill 1407.26 ·
- Ollama · Q4_K_M Decode 345.26 · Prefill 1726.58 ·
- Strata · Q4_K_M Decode 317.23 · Prefill 1729.64 ·
- vLLM · IQ2_M Decode 325.98 · Prefill 1095.54 ·
- llama.cpp · Q6_K Decode 343.53 · Prefill 1745.18 ·
- Ollama · Q3_K_S Decode 313.56 · Prefill 1182.08 ·
- llama.cpp · Q4_K_S Decode 342.54 · Prefill 1113.53 ·
- Strata · Q3_K_S Decode 345.96 · Prefill 1251.15 ·
- Ollama · Q3_K_M Decode 303.51 · Prefill 1532.77 ·
- Strata · IQ3_S Decode 322.41 · Prefill 1596.98 ·
- vLLM · Q4_K_M Decode 224.13 · Prefill 1208.86 ·
- vLLM · Q5_K_M Decode 206.52 · Prefill 1029.93 ·
- Strata · IQ2_M Decode 325.92 · Prefill 1772.8 ·
- vLLM · Q8_0 Decode 234.17 · Prefill 755.94 ·
- Strata · Q3_K_S Decode 335.33 · Prefill 1692.56 ·
- Strata · Q8_0 Decode 297.87 · Prefill 1046.41 ·
- vLLM · IQ4_XS Decode 288.72 · Prefill 1282.53 ·
- vLLM · Q3_K_M Decode 355.61 · Prefill 1722.65 ·
- Ollama · IQ2_M Decode 321.92 · Prefill 1762.51 ·
- Ollama · Q8_0 Decode 296.95 · Prefill 1140.84 ·
- vLLM · IQ3_S Decode 329.61 · Prefill 1536.04 ·
- vLLM · Q3_K_S Decode 362.27 · Prefill 1696.48 ·
- Strata · Q2_K Decode 373.13 · Prefill 1907.75 ·
- llama.cpp · Q4_K_S Decode 340.19 · Prefill 1569.8 ·
- vLLM · IQ2_M Decode 294.68 · Prefill 1324.31 ·
- vLLM · Q3_K_S Decode 355.68 · Prefill 1952.96 ·
- llama.cpp · IQ3_XS Decode 324.47 · Prefill 1251.53 ·
- Ollama · IQ3_XS Decode 374.78 · Prefill 1868.44 ·
- vLLM · IQ1_S Decode 335.51 · Prefill 1078.48 ·
- Ollama · Q6_K Decode 301.71 · Prefill 1324.79 ·
- vLLM · IQ3_XS Decode 338.72 · Prefill 1535.63 ·
- vLLM · Q3_K_M Decode 275.73 · Prefill 1508 ·
- Strata · Q5_K_S Decode 299.9 · Prefill 1290.71 ·
- Strata · Q6_K Decode 276.83 · Prefill 1508.76 ·
- llama.cpp · Q5_K_M Decode 303.32 · Prefill 1265.92 ·
- llama.cpp · Q5_K_S Decode 283.07 · Prefill 1445.04 ·
- vLLM · IQ3_XS Decode 302.76 · Prefill 1418.76 ·
- vLLM · IQ4_XS Decode 358.44 · Prefill 1162.82 ·
- llama.cpp · IQ1_S Decode 337.53 · Prefill 1795.74 ·
- llama.cpp · Q4_K_S Decode 314.55 · Prefill 1364.71 ·
- Strata · Q5_K_M Decode 362.58 · Prefill 1954.02 ·
- llama.cpp · IQ3_S Decode 289.19 · Prefill 992.46 ·
- llama.cpp · IQ4_XS Decode 283.18 · Prefill 981.02 ·
- vLLM · Q2_K Decode 349.23 · Prefill 1726.15 ·
- llama.cpp · FP16 Decode 379.95 · Prefill 1874.2 ·
Fit 判定
按显存与内存带宽做物理估算:选择模型,查看各量化能否装下及预计速度
估算公式:tok/s =(带宽 ÷ 模型占用)× 0.55,借鉴 llmfit;实测以运行记录为准
核心规格
- 厂商
- NVIDIA
- 显存
- 80 GB
- 显存带宽
- 3350 GB/s
运行记录
共 50 组配置
| 模型 | 框架 | 量化 | Decode | 生成耗时 | 样本数 |
|---|---|---|---|---|---|
| gemma-4-E4B-it | llama.cpp | IQ3_XS | 297.4 tok/s | — | 1 |
| gemma-4-E4B-it | vLLM | Q3_K_S | 343.3 tok/s | — | 1 |
| Ministral-3-8B-Instruct-2512 | llama.cpp | Q8_0 | 271.7 tok/s | — | 1 |
| Ministral-3-8B-Instruct-2512 | vLLM | Q3_K_M | 321.7 tok/s | — | 1 |
| iFlytek Spark-X2.5-4B | Ollama | IQ2_XS | 285.6 tok/s | — | 1 |
| iFlytek Spark-X2.5-4B | Strata | Q3_K_S | 291.7 tok/s | — | 1 |
| NVIDIA-Nemotron-3.5-Bolt-E4B-4GB | Ollama | Q4_K_M | 345.3 tok/s | — | 1 |
| NVIDIA-Nemotron-3.5-Bolt-E4B-4GB | Strata | Q4_K_M | 317.2 tok/s | — | 1 |
| Flint-40B-A2.5B | vLLM | IQ2_M | 326 tok/s | — | 1 |
| Raon-1-mini-preview | llama.cpp | Q6_K | 343.5 tok/s | — | 1 |
| Raon-1-mini-preview | Ollama | Q3_K_S | 313.6 tok/s | — | 1 |
| mammoth-1 | llama.cpp | Q4_K_S | 342.5 tok/s | — | 1 |
| mammoth-1 | Strata | Q3_K_S | 346 tok/s | — | 1 |
| elephant-1 | Ollama | Q3_K_M | 303.5 tok/s | — | 1 |
| elephant-1 | Strata | IQ3_S | 322.4 tok/s | — | 1 |
| Qwen3.8-27B | vLLM | Q4_K_M | 224.1 tok/s | — | 1 |
| Qwen3.8-27B | vLLM | Q5_K_M | 206.5 tok/s | — | 1 |
| Qwen3.8-27B | Strata | IQ2_M | 325.9 tok/s | — | 1 |
| Qwen3.6-27B | vLLM | Q8_0 | 234.2 tok/s | — | 1 |
| Qwen3.6-27B | Strata | Q3_K_S | 335.3 tok/s | — | 1 |
| Qwen3.6-27B | Strata | Q8_0 | 297.9 tok/s | — | 1 |
| Qwen3.6-35B-A3B | vLLM | IQ4_XS | 288.7 tok/s | — | 1 |
| Qwen3.6-35B-A3B | vLLM | Q3_K_M | 355.6 tok/s | — | 1 |
| Qwen3.6-35B-A3B | Ollama | IQ2_M | 321.9 tok/s | — | 1 |
| Qwen3.6-35B-A3B | Ollama | Q8_0 | 296.9 tok/s | — | 1 |
| Qwen3.5-4B | vLLM | IQ3_S | 329.6 tok/s | — | 1 |
| Qwen3.5-4B | vLLM | Q3_K_S | 362.3 tok/s | — | 1 |
| Qwen3.5-4B | Strata | Q2_K | 373.1 tok/s | — | 1 |
| Qwen3.5-0.8B | llama.cpp | Q4_K_S | 340.2 tok/s | — | 1 |
| Qwen3.5-0.8B | vLLM | IQ2_M | 294.7 tok/s | — | 1 |
| Qwen3.5-0.8B | vLLM | Q3_K_S | 355.7 tok/s | — | 1 |
| Qwen3.8-9B | llama.cpp | IQ3_XS | 324.5 tok/s | — | 1 |
| Qwen3.8-9B | Ollama | IQ3_XS | 374.8 tok/s | — | 1 |
| Qwen3-8B | vLLM | IQ1_S | 335.5 tok/s | — | 1 |
| Qwen3-8B | Ollama | Q6_K | 301.7 tok/s | — | 1 |
| gemma-4-31B-it | vLLM | IQ3_XS | 338.7 tok/s | — | 1 |
| gemma-4-31B-it | vLLM | Q3_K_M | 275.7 tok/s | — | 1 |
| gemma-4-31B-it | Strata | Q5_K_S | 299.9 tok/s | — | 1 |
| gemma-4-31B-it | Strata | Q6_K | 276.8 tok/s | — | 1 |
| gemma-4-12B-it | llama.cpp | Q5_K_M | 303.3 tok/s | — | 1 |
| gemma-4-12B-it | llama.cpp | Q5_K_S | 283.1 tok/s | — | 1 |
| gemma-4-12B-it | vLLM | IQ3_XS | 302.8 tok/s | — | 1 |
| gemma-4-12B-it | vLLM | IQ4_XS | 358.4 tok/s | — | 1 |
| Llama-3.1-8B-Instruct | llama.cpp | IQ1_S | 337.5 tok/s | — | 1 |
| Llama-3.1-8B-Instruct | llama.cpp | Q4_K_S | 314.6 tok/s | — | 1 |
| Llama-3.1-8B-Instruct | Strata | Q5_K_M | 362.6 tok/s | — | 1 |
| Llama-3-8B-Instruct | llama.cpp | IQ3_S | 289.2 tok/s | — | 1 |
| Llama-3-8B-Instruct | llama.cpp | IQ4_XS | 283.2 tok/s | — | 1 |
| Llama-3-8B-Instruct | vLLM | Q2_K | 349.2 tok/s | — | 1 |
| Llama-3.2-3B-Instruct | llama.cpp | FP16 | 379.9 tok/s | — | 1 |