Qwen3-8B · Q4_K_M
L2 Cross-framework verifiedNVIDIA RTX 4090 · 24GB · llama.cpp
Other
2026-10-19
142.71 tok/s
Decode
813.73 tok/s
Prefill
1.2 s
TTFT
10.64 GB
VRAM
Config (latest run)
Sources: Other · 2 runs · 1 independent sources
Each row is a run configuration (model + hardware + framework + quantization); repeated runs of the same setup are merged — raw counts appear in each result’s source line.
NVIDIA RTX 4090 · 24GB · llama.cpp
Other
2026-10-19
142.71 tok/s
Decode
813.73 tok/s
Prefill
1.2 s
TTFT
10.64 GB
VRAM
Sources: Other · 2 runs · 1 independent sources
NVIDIA RTX 4090 · 24GB · vLLM
Other
2026-10-19
161.14 tok/s
Decode
813.02 tok/s
Prefill
1.76 s
TTFT
13.19 GB
VRAM
Sources: Other · 1 runs · 1 independent sources
NVIDIA RTX 3090 · 24GB · vLLM
Other
2026-10-19
113.39 tok/s
Decode
440.93 tok/s
Prefill
2.12 s
TTFT
10.16 GB
VRAM
Sources: Other · 1 runs · 1 independent sources
NVIDIA RTX 3090 · 24GB · Ollama
Other
2026-10-19
98.87 tok/s
Decode
381.75 tok/s
Prefill
0.26 s
TTFT
11.34 GB
VRAM
Sources: Other · 1 runs · 1 independent sources
AMD Radeon RX 7900 XTX · 24GB · llama.cpp
Other
2026-10-19
73.41 tok/s
Decode
264.95 tok/s
Prefill
1.27 s
TTFT
17.22 GB
VRAM
Sources: Other · 1 runs · 1 independent sources
AMD Radeon RX 7900 XTX · 24GB · vLLM
Other
2026-10-19
127.88 tok/s
Decode
585.23 tok/s
Prefill
0.17 s
TTFT
10.84 GB
VRAM
Sources: Other · 2 runs · 1 independent sources
AMD Radeon RX 7900 XTX · 24GB · vLLM
Other
2026-10-19
108.21 tok/s
Decode
417.97 tok/s
Prefill
1.3 s
TTFT
11.34 GB
VRAM
Sources: Other · 1 runs · 1 independent sources
NVIDIA RTX 5090 · 32GB · Ollama
Other
2026-10-19
275.39 tok/s
Decode
1,404.89 tok/s
Prefill
2.09 s
TTFT
13.19 GB
VRAM
Sources: Other · 1 runs · 1 independent sources
NVIDIA RTX 5090 · 32GB · Strata
Other
2026-10-19
340.69 tok/s
Decode
1,198.68 tok/s
Prefill
0.14 s
TTFT
14.53 GB
VRAM
Sources: Other · 1 runs · 1 independent sources
NVIDIA RTX 5080 · 16GB · llama.cpp
Other
2026-10-19
206.58 tok/s
Decode
831.19 tok/s
Prefill
0.14 s
TTFT
13.02 GB
VRAM
Sources: Other · 1 runs · 1 independent sources
NVIDIA RTX 5080 · 16GB · vLLM
Other
2026-10-19
240.94 tok/s
Decode
1,234.91 tok/s
Prefill
0.25 s
TTFT
10.84 GB
VRAM
Sources: Other · 1 runs · 1 independent sources
NVIDIA RTX 3060 12GB · 12GB · vLLM
Other
2026-10-19
73.32 tok/s
Decode
238.24 tok/s
Prefill
0.61 s
TTFT
11.4 GB
VRAM
Sources: Other · 1 runs · 1 independent sources
NVIDIA RTX 3060 12GB · 12GB · Ollama
Other
2026-10-19
58.42 tok/s
Decode
197.59 tok/s
Prefill
1.92 s
TTFT
11.4 GB
VRAM
Sources: Other · 1 runs · 1 independent sources
NVIDIA RTX 4080 Super · 16GB · llama.cpp
Other
2026-10-19
168.36 tok/s
Decode
655.5 tok/s
Prefill
1.91 s
TTFT
13.02 GB
VRAM
Sources: Other · 1 runs · 1 independent sources
NVIDIA RTX 4080 Super · 16GB · vLLM
Other
2026-10-19
214.44 tok/s
Decode
870.17 tok/s
Prefill
0.98 s
TTFT
10.84 GB
VRAM
Sources: Other · 2 runs · 1 independent sources
AMD Ryzen AI Max+ 395 · 128GB · vLLM
Other
2026-10-19
38.79 tok/s
Decode
186.13 tok/s
Prefill
0.28 s
TTFT
13.02 GB
VRAM
Sources: Other · 1 runs · 1 independent sources
AMD Ryzen AI Max+ 395 · 128GB · Ollama
Other
2026-10-19
14.12 tok/s
Decode
69.25 tok/s
Prefill
0.35 s
TTFT
33.18 GB
VRAM
Sources: Other · 1 runs · 1 independent sources
AMD Ryzen AI Max+ 395 · 128GB · Ollama
Other
2026-10-19
34.18 tok/s
Decode
132.79 tok/s
Prefill
0.58 s
TTFT
13.52 GB
VRAM
Sources: Other · 1 runs · 1 independent sources
Apple M5 Max · 128GB · llama.cpp
Other
2026-10-19
95.3 tok/s
Decode
467.44 tok/s
Prefill
1.9 s
TTFT
9.16 GB
VRAM
Sources: Other · 1 runs · 1 independent sources
Apple M5 Max · 128GB · Ollama
Other
2026-10-19
82.41 tok/s
Decode
283.9 tok/s
Prefill
2.19 s
TTFT
10.84 GB
VRAM
Sources: Other · 1 runs · 1 independent sources
199 results