Benchmark search

Try these searches

Found 102 run configurations

Each row is a run configuration (model + hardware + framework + quantization); repeated runs of the same setup are merged — raw counts appear in each result’s source line.

ModelQwen3-4B-Thinking-2507

Qwen3-4B-Thinking-2507 · Q2_K

L2 Cross-framework verified

NVIDIA RTX 5090 · 32GB · llama.cpp

Other

2026-10-15

392.48 tok/s

Decode

1,604.11 tok/s

Prefill

0.17 s

TTFT

6.72 GB

VRAM

Config (latest run)
Ubuntu 24.04 CUDA 12.8 Q2_K llama.cpp v0.33

Sources: Other · 1 runs · 1 independent sources

Qwen3-4B-Thinking-2507 · Q8_0

L2 Cross-framework verified

NVIDIA RTX 5090 · 32GB · llama.cpp

Other

2026-10-15

184.55 tok/s

Decode

803.14 tok/s

Prefill

1.75 s

TTFT

11.76 GB

VRAM

Config (latest run)
Ubuntu 22.04 CUDA 12.4 Q8_0 llama.cpp v0.47 Flash Attention

Sources: Other · 1 runs · 1 independent sources

Qwen3-4B-Thinking-2507 · IQ2_M

L2 Cross-framework verified

NVIDIA RTX 5090 · 32GB · vLLM

Other

2026-10-15

386.87 tok/s

Decode

1,842.03 tok/s

Prefill

2.34 s

TTFT

6.47 GB

VRAM

Config (latest run)
Ubuntu 24.04 CUDA 12.8 IQ2_M vLLM v0.49

Sources: Other · 1 runs · 1 independent sources

Qwen3-4B-Thinking-2507 · Q3_K_L

L2 Cross-framework verified

NVIDIA RTX 5090 · 32GB · vLLM

Other

2026-10-15

337.16 tok/s

Decode

1,247.18 tok/s

Prefill

0.59 s

TTFT

8.15 GB

VRAM

Config (latest run)
Ubuntu 24.04 CUDA 12.4 Q3_K_L vLLM v0.43 Flash Attention

Sources: Other · 1 runs · 1 independent sources

Qwen3-4B-Thinking-2507 · Q5_K_M

L2 Cross-framework verified

AMD Ryzen AI Max+ 395 · 128GB · llama.cpp

Other

2026-10-15

107.36 tok/s

Decode

478 tok/s

Prefill

0.7 s

TTFT

9.91 GB

VRAM

Config (latest run)
Ubuntu 22.04 CUDA N/A Q5_K_M llama.cpp v0.47 Flash Attention

Sources: Other · 2 runs · 1 independent sources

Qwen3-4B-Thinking-2507 · Q8_0

L2 Cross-framework verified

AMD Ryzen AI Max+ 395 · 128GB · llama.cpp

Other

2026-10-15

78.29 tok/s

Decode

263.52 tok/s

Prefill

0.96 s

TTFT

11.76 GB

VRAM

Config (latest run)
macOS 15.0 CUDA N/A Q8_0 llama.cpp v0.51

Sources: Other · 1 runs · 1 independent sources

Qwen3-4B-Thinking-2507 · Q3_K_S

L2 Cross-framework verified

AMD Ryzen AI Max+ 395 · 128GB · Ollama

Other

2026-10-15

145.01 tok/s

Decode

581.64 tok/s

Prefill

2 s

TTFT

7.56 GB

VRAM

Config (latest run)
macOS 15.0 CUDA N/A Q3_K_S Ollama v0.53

Sources: Other · 1 runs · 1 independent sources

Qwen3-4B-Thinking-2507 · Q5_K_S

L2 Cross-framework verified

Apple M5 Max · 128GB · Ollama

Other

2026-10-15

192.3 tok/s

Decode

1,031.35 tok/s

Prefill

0.19 s

TTFT

9.66 GB

VRAM

Config (latest run)
Ubuntu 22.04 CUDA 12.6 Q5_K_S Ollama v0.54 Flash Attention

Sources: Other · 1 runs · 1 independent sources

Qwen3-4B-Thinking-2507 · Q5_K_M

L2 Cross-framework verified

Apple M5 Max · 128GB · Strata

Other

2026-10-15

281.24 tok/s

Decode

1,191.19 tok/s

Prefill

0.52 s

TTFT

9.91 GB

VRAM

Config (latest run)
macOS 15.0 CUDA 12.6 Q5_K_M Strata v0.40 Flash Attention

Sources: Other · 1 runs · 1 independent sources

Qwen3-4B-Thinking-2507 · IQ1_S

L2 Cross-framework verified

NVIDIA DGX Spark · 128GB · vLLM

Other

2026-10-15

291.5 tok/s

Decode

1,360.04 tok/s

Prefill

1.03 s

TTFT

5.63 GB

VRAM

Config (latest run)
Ubuntu 24.04 CUDA 12.4 IQ1_S vLLM v0.33 Flash Attention

Sources: Other · 1 runs · 1 independent sources

Qwen3-4B-Thinking-2507 · Q5_K_M

L2 Cross-framework verified

NVIDIA DGX Spark · 128GB · vLLM

Other

2026-10-15

151.71 tok/s

Decode

832.95 tok/s

Prefill

2.16 s

TTFT

9.91 GB

VRAM

Config (latest run)
Ubuntu 22.04 CUDA 12.4 Q5_K_M vLLM v0.55 Flash Attention

Sources: Other · 1 runs · 1 independent sources

Qwen3-4B-Thinking-2507 · Q5_K_M

L2 Cross-framework verified

NVIDIA DGX Spark · 128GB · Strata

Other

2026-10-15

220.61 tok/s

Decode

780.86 tok/s

Prefill

1.84 s

TTFT

9.91 GB

VRAM

Config (latest run)
Ubuntu 22.04 CUDA 12.8 Q5_K_M Strata v0.54 Flash Attention

Sources: Other · 1 runs · 1 independent sources

Qwen3-4B-Thinking-2507 · Q3_K_S

L2 Cross-framework verified

NVIDIA Jetson AGX Orin 64GB · 64GB · vLLM

Other

2026-10-15

89.8 tok/s

Decode

485.57 tok/s

Prefill

1.3 s

TTFT

7.56 GB

VRAM

Config (latest run)
JetPack 6.0 CUDA 12.6 Q3_K_S vLLM v0.52

Sources: Other · 1 runs · 1 independent sources

Qwen3-4B-Thinking-2507 · Q5_K_S

L2 Cross-framework verified

NVIDIA Jetson AGX Orin 64GB · 64GB · vLLM

Other

2026-10-15

57.15 tok/s

Decode

278.01 tok/s

Prefill

1.97 s

TTFT

9.66 GB

VRAM

Config (latest run)
JetPack 6.0 CUDA 12.6 Q5_K_S vLLM v0.26

Sources: Other · 1 runs · 1 independent sources

Qwen3-4B-Thinking-2507 · Q2_K

L2 Cross-framework verified

NVIDIA Jetson AGX Orin 64GB · 64GB · Ollama

Other

2026-10-15

77.04 tok/s

Decode

283.31 tok/s

Prefill

1.72 s

TTFT

6.72 GB

VRAM

Config (latest run)
JetPack 6.0 CUDA 12.6 Q2_K Ollama v0.54 Flash Attention

Sources: Other · 1 runs · 1 independent sources

Qwen3-4B-Thinking-2507 · Q3_K_S

L2 Cross-framework verified

AMD Strix Halo (Radeon 8060S) · llama.cpp

Other

2026-10-15

117.04 tok/s

Decode

404.22 tok/s

Prefill

1.25 s

TTFT

7.56 GB

VRAM

Config (latest run)
Windows 11 23H2 CUDA N/A Q3_K_S llama.cpp v0.26

Sources: Other · 1 runs · 1 independent sources

Qwen3-4B-Thinking-2507 · Q5_K_M

L2 Cross-framework verified

AMD Strix Halo (Radeon 8060S) · llama.cpp

Other

2026-10-15

77.99 tok/s

Decode

423.17 tok/s

Prefill

0.35 s

TTFT

9.91 GB

VRAM

Config (latest run)
Ubuntu 24.04 CUDA N/A Q5_K_M llama.cpp v0.39 Flash Attention

Sources: Other · 1 runs · 1 independent sources

Qwen3-4B-Thinking-2507 · IQ2_XS

L2 Cross-framework verified

AMD Strix Halo (Radeon 8060S) · Strata

Other

2026-10-15

206.18 tok/s

Decode

1,005.37 tok/s

Prefill

1.09 s

TTFT

6.13 GB

VRAM

Config (latest run)
Ubuntu 22.04 CUDA N/A IQ2_XS Strata v0.47

Sources: Other · 1 runs · 1 independent sources

Qwen3-4B-Thinking-2507 · IQ3_S

L2 Cross-framework verified

AMD Strix Halo (Radeon 8060S) · Strata

Other

2026-10-15

166.26 tok/s

Decode

669.87 tok/s

Prefill

0.77 s

TTFT

7.64 GB

VRAM

Config (latest run)
Ubuntu 24.04 CUDA N/A IQ3_S Strata v0.28 Flash Attention

Sources: Other · 1 runs · 1 independent sources

Qwen3-4B-Thinking-2507 · Q3_K_M

L2 Cross-framework verified

2x AMD Strix Halo · llama.cpp

Other

2026-10-15

181.31 tok/s

Decode

944.53 tok/s

Prefill

2.23 s

TTFT

7.81 GB

VRAM

Config (latest run)
macOS 15.0 CUDA N/A Q3_K_M llama.cpp v0.24 Flash Attention

Sources: Other · 1 runs · 1 independent sources

102 results

Related models
Related hardware
Related frameworks
Related quantizations