Arao AI

Benchmark search

Try these searches

Found 95 run configurations

Each row is a run configuration (model + hardware + framework + quantization); repeated runs of the same setup are merged — raw counts appear in each result’s source line.

ModelLlama-3.1-8B-Instruct

Llama-3.1-8B-Instruct · Q4_K_M

L2 Cross-framework verified

NVIDIA RTX Spark · 128GB · llama.cpp

Other

16 days ago

53.5 tok/s

Decode

208.91 tok/s

Prefill

1.72 s

TTFT

15.88 GB

VRAM

Config (latest run)
Ubuntu 24.04 CUDA 13.0 Q4_K_M llama.cpp v0.46

Sources: Other · 2 runs · 1 independent sources

View details

Llama-3.1-8B-Instruct · Q5_K_M

L2 Cross-framework verified

NVIDIA RTX Spark · 128GB · llama.cpp

Other

16 days ago

46.73 tok/s

Decode

200.62 tok/s

Prefill

2.34 s

TTFT

17.72 GB

VRAM

Config (latest run)
Ubuntu 24.04 CUDA 13.0 Q5_K_M llama.cpp v0.25 Flash Attention

Sources: Other · 1 runs · 1 independent sources

View details

Llama-3.1-8B-Instruct · Q8_0

L2 Cross-framework verified

NVIDIA RTX Spark · 128GB · llama.cpp

Other

16 days ago

35.36 tok/s

Decode

182.04 tok/s

Prefill

1.26 s

TTFT

21.42 GB

VRAM

Config (latest run)
Ubuntu 24.04 CUDA 13.0 Q8_0 llama.cpp v0.33

Sources: Other · 1 runs · 1 independent sources

View details

Llama-3.1-8B-Instruct · FP16

L2 Cross-framework verified

NVIDIA RTX Spark · 128GB · Ollama

Other

16 days ago

27.91 tok/s

Decode

99.5 tok/s

Prefill

0.97 s

TTFT

33.18 GB

VRAM

Config (latest run)
Ubuntu 24.04 CUDA 13.0 FP16 Ollama v0.46 Flash Attention

Sources: Other · 1 runs · 1 independent sources

View details

Llama-3.1-8B-Instruct · Q6_K

L2 Cross-framework verified

NVIDIA RTX Spark · 128GB · Ollama

Other

16 days ago

40.65 tok/s

Decode

194.34 tok/s

Prefill

2.58 s

TTFT

18.9 GB

VRAM

Config (latest run)
Ubuntu 24.04 CUDA 13.0 Q6_K Ollama v0.43

Sources: Other · 1 runs · 1 independent sources

View details

Llama-3.1-8B-Instruct · Q8_0

L2 Cross-framework verified

NVIDIA RTX Spark · 128GB · Ollama

Other

16 days ago

37 tok/s

Decode

131.55 tok/s

Prefill

1.13 s

TTFT

21.42 GB

VRAM

Config (latest run)
Ubuntu 24.04 CUDA 13.0 Q8_0 Ollama v0.48

Sources: Other · 1 runs · 1 independent sources

View details

Llama-3.1-8B-Instruct · IQ2_XS

L2 Cross-framework verified

NVIDIA RTX 4090 · 24GB · vLLM

Other

17 days ago

205.12 tok/s

Decode

1,126.08 tok/s

Prefill

1.81 s

TTFT

10.16 GB

VRAM

Config (latest run)
Windows 11 24H2 CUDA 12.4 IQ2_XS vLLM v0.31 Flash Attention

Sources: Other · 1 runs · 1 independent sources

View details

Llama-3.1-8B-Instruct · Q4_K_M

L2 Cross-framework verified

NVIDIA RTX 4090 · 24GB · Strata

Other

17 days ago

191.74 tok/s

Decode

642.2 tok/s

Prefill

1.28 s

TTFT

15.88 GB

VRAM

Config (latest run)
Windows 11 23H2 CUDA 12.4 Q4_K_M Strata v0.29 Flash Attention

Sources: Other · 1 runs · 1 independent sources

View details

Llama-3.1-8B-Instruct · Q6_K

L2 Cross-framework verified

NVIDIA RTX 4090 · 24GB · Strata

Other

17 days ago

151.86 tok/s

Decode

574.29 tok/s

Prefill

2.15 s

TTFT

18.9 GB

VRAM

Config (latest run)
Ubuntu 24.04 CUDA 12.6 Q6_K Strata v0.37 Flash Attention

Sources: Other · 1 runs · 1 independent sources

View details

Llama-3.1-8B-Instruct · Q8_0

L2 Cross-framework verified

NVIDIA RTX 3090 · 24GB · vLLM

Other

17 days ago

64.44 tok/s

Decode

290.44 tok/s

Prefill

1.31 s

TTFT

21.42 GB

VRAM

Config (latest run)
Windows 11 24H2 CUDA 12.8 Q8_0 vLLM v0.54 Flash Attention

Sources: Other · 1 runs · 1 independent sources

View details

Llama-3.1-8B-Instruct · IQ3_XS

L2 Cross-framework verified

NVIDIA RTX 3090 · 24GB · Ollama

Other

17 days ago

101.05 tok/s

Decode

370.05 tok/s

Prefill

1.79 s

TTFT

12.52 GB

VRAM

Config (latest run)
Windows 11 23H2 CUDA 12.6 IQ3_XS Ollama v0.29

Sources: Other · 1 runs · 1 independent sources

View details

Llama-3.1-8B-Instruct · Q3_K_S

L2 Cross-framework verified

AMD Radeon RX 7900 XTX · 24GB · llama.cpp

Other

17 days ago

102.17 tok/s

Decode

424.1 tok/s

Prefill

1.7 s

TTFT

13.02 GB

VRAM

Config (latest run)
Windows 11 23H2 CUDA 12.8 Q3_K_S llama.cpp v0.39 Flash Attention

Sources: Other · 1 runs · 1 independent sources

View details

Llama-3.1-8B-Instruct · Q4_K_M

L2 Cross-framework verified

AMD Radeon RX 7900 XTX · 24GB · llama.cpp

Other

17 days ago

83.47 tok/s

Decode

411.3 tok/s

Prefill

1.84 s

TTFT

15.88 GB

VRAM

Config (latest run)
Ubuntu 22.04 CUDA 12.4 Q4_K_M llama.cpp v0.20 Flash Attention

Sources: Other · 1 runs · 1 independent sources

View details

Llama-3.1-8B-Instruct · Q3_K_M

L2 Cross-framework verified

AMD Radeon RX 7900 XTX · 24GB · vLLM

Other

17 days ago

114.54 tok/s

Decode

374.54 tok/s

Prefill

1.68 s

TTFT

13.52 GB

VRAM

Config (latest run)
Windows 11 23H2 CUDA 12.8 Q3_K_M vLLM v0.21 Flash Attention

Sources: Other · 1 runs · 1 independent sources

View details

Llama-3.1-8B-Instruct · Q4_K_M

L2 Cross-framework verified

AMD Radeon RX 7900 XTX · 24GB · vLLM

Other

17 days ago

87.13 tok/s

Decode

426.78 tok/s

Prefill

1.46 s

TTFT

15.88 GB

VRAM

Config (latest run)
Ubuntu 24.04 CUDA 12.8 Q4_K_M vLLM v0.32 Flash Attention

Sources: Other · 1 runs · 1 independent sources

View details

Llama-3.1-8B-Instruct · Q5_K_S

L2 Cross-framework verified

NVIDIA RTX 5090 · 32GB · Ollama

Other

17 days ago

220.5 tok/s

Decode

854 tok/s

Prefill

1.41 s

TTFT

17.22 GB

VRAM

Config (latest run)
Windows 11 23H2 CUDA 12.4 Q5_K_S Ollama v0.23

Sources: Other · 1 runs · 1 independent sources

View details

Llama-3.1-8B-Instruct · IQ3_S

L2 Cross-framework verified

NVIDIA RTX 5090 · 32GB · Strata

Other

17 days ago

280 tok/s

Decode

1,358.66 tok/s

Prefill

1.41 s

TTFT

13.19 GB

VRAM

Config (latest run)
Ubuntu 24.04 CUDA 12.6 IQ3_S Strata v0.22 Flash Attention

Sources: Other · 1 runs · 1 independent sources

View details

Llama-3.1-8B-Instruct · Q5_K_M

L2 Cross-framework verified

NVIDIA RTX 5090 · 32GB · Strata

Other

17 days ago

297.96 tok/s

Decode

1,229.8 tok/s

Prefill

2.28 s

TTFT

17.72 GB

VRAM

Config (latest run)
Ubuntu 22.04 CUDA 12.8 Q5_K_M Strata v0.21 Flash Attention

Sources: Other · 1 runs · 1 independent sources

View details

Llama-3.1-8B-Instruct · Q5_K_S

L2 Cross-framework verified

NVIDIA RTX 5080 · 16GB · Ollama

Other

17 days ago

149.47 tok/s

Decode

563.58 tok/s

Prefill

1.49 s

TTFT

15.2 GB

VRAM

Config (latest run)
Ubuntu 22.04 CUDA 12.4 Q5_K_S Ollama v0.21 Flash Attention

Sources: Other · 1 runs · 1 independent sources

View details

Llama-3.1-8B-Instruct · IQ3_S

L2 Cross-framework verified

NVIDIA RTX 5080 · 16GB · Strata

Other

17 days ago

302.96 tok/s

Decode

1,550.76 tok/s

Prefill

2.09 s

TTFT

13.19 GB

VRAM

Config (latest run)
Windows 11 23H2 CUDA 12.4 IQ3_S Strata v0.26

Sources: Other · 1 runs · 1 independent sources

View details

95 results

Related models
Related hardware
Related frameworks
Related quantizations