Llama-3-8B-Instruct

8BChatby Meta Start a discussion

0

Quantized builds

91

Benchmark records

Quantized builds

Community and official quantized releases; "linked records" counts only benchmarks tied to that exact release.

No quantized builds cataloged yet.

Benchmark data

50 configurations

HardwareFrameworkQuantDecodeVRAM usedSamplesEvidence
NVIDIA RTX 4090llama.cppQ5_K_S118.7 tok/s17.22 GB1L2 Cross-framework verified
NVIDIA RTX 4090vLLMIQ3_S177.4 tok/s13.19 GB1L2 Cross-framework verified
NVIDIA RTX 4090vLLMQ4_K_S157.4 tok/s15.2 GB1L2 Cross-framework verified
NVIDIA RTX 3090OllamaQ5_K_S68 tok/s17.22 GB1L2 Cross-framework verified
NVIDIA RTX 3090StrataIQ3_S155.9 tok/s13.19 GB1L2 Cross-framework verified
NVIDIA RTX 3090StrataQ5_K_M101.9 tok/s17.72 GB1L2 Cross-framework verified
AMD Radeon RX 7900 XTXvLLMQ4_K_M95.7 tok/s15.88 GB1L2 Cross-framework verified
AMD Radeon RX 7900 XTXOllamaQ5_K_S74.7 tok/s17.22 GB2L2 Cross-framework verified
NVIDIA RTX 5090vLLMIQ3_XS353.4 tok/s12.52 GB2L2 Cross-framework verified
NVIDIA RTX 5090vLLMQ4_K_M288.4 tok/s15.88 GB1L2 Cross-framework verified
NVIDIA RTX 5090OllamaQ5_K_M226.3 tok/s17.72 GB1L2 Cross-framework verified
NVIDIA RTX 5080OllamaIQ3_S199.2 tok/s13.19 GB1L2 Cross-framework verified
NVIDIA RTX 5080OllamaQ3_K_S204.5 tok/s13.02 GB2L2 Cross-framework verified
NVIDIA RTX 5080StrataIQ1_S349.7 tok/s9.16 GB1L2 Cross-framework verified
NVIDIA RTX 5080StrataQ3_K_L297.9 tok/s14.2 GB1L2 Cross-framework verified
NVIDIA RTX 3060 12GBStrataQ3_K_M95.2 tok/s11.4 GB1L2 Cross-framework verified
NVIDIA RTX 4080 SuperOllamaIQ3_XS190.9 tok/s12.52 GB2L2 Cross-framework verified
NVIDIA RTX 4080 SuperStrataIQ2_XS334.5 tok/s10.16 GB1L2 Cross-framework verified
AMD Ryzen AI Max+ 395vLLMIQ2_M48.9 tok/s10.84 GB1L2 Cross-framework verified
AMD Ryzen AI Max+ 395vLLMQ5_K_M31.8 tok/s17.72 GB1L2 Cross-framework verified
AMD Ryzen AI Max+ 395OllamaQ3_K_L37.4 tok/s14.2 GB1L2 Cross-framework verified
Apple M5 Maxllama.cppQ3_K_M83.6 tok/s13.52 GB1L2 Cross-framework verified
Apple M5 MaxOllamaIQ2_M100 tok/s10.84 GB1L2 Cross-framework verified
Apple M5 Ultrallama.cppIQ2_M145.5 tok/s10.84 GB1L2 Cross-framework verified
Apple M5 Ultrallama.cppQ4_K_S115.5 tok/s15.2 GB1L2 Cross-framework verified
Apple M5 UltraOllamaIQ3_XS140.7 tok/s12.52 GB1L2 Cross-framework verified
NVIDIA H100 80GB HBM3llama.cppIQ3_S289.2 tok/s13.19 GB1L2 Cross-framework verified
NVIDIA H100 80GB HBM3llama.cppIQ4_XS283.2 tok/s14.53 GB1L2 Cross-framework verified
NVIDIA H100 80GB HBM3vLLMQ2_K349.2 tok/s11.34 GB1L2 Cross-framework verified
NVIDIA H20vLLMQ2_K77.7 tok/s11.34 GB1L2 Cross-framework verified
NVIDIA H20OllamaQ5_K_M41.5 tok/s17.72 GB1L2 Cross-framework verified
NVIDIA H20OllamaQ8_036.5 tok/s21.42 GB1L2 Cross-framework verified
NVIDIA RTX PRO 6000 Blackwell 96GBllama.cppIQ2_M345.7 tok/s10.84 GB1L2 Cross-framework verified
NVIDIA RTX PRO 6000 Blackwell 96GBllama.cppQ5_K_M194.4 tok/s17.72 GB1L2 Cross-framework verified
NVIDIA RTX PRO 6000 Blackwell 96GBOllamaQ3_K_M249.2 tok/s13.52 GB1L2 Cross-framework verified
NVIDIA RTX PRO 6000 Blackwell 96GBOllamaQ6_K185.7 tok/s18.9 GB1L2 Cross-framework verified
Intel Arc B580llama.cppQ3_K_M57.8 tok/s11.4 GB1L2 Cross-framework verified
Intel Arc B580OllamaQ2_K67 tok/s11.34 GB1L2 Cross-framework verified
Apple M2 Ultrallama.cppIQ2_M144 tok/s10.84 GB1L2 Cross-framework verified
Apple M2 Ultrallama.cppQ2_K151.5 tok/s11.34 GB1L2 Cross-framework verified
Apple M2 UltraOllamaIQ3_S116.9 tok/s13.19 GB1L2 Cross-framework verified
Apple M2 UltraOllamaQ3_K_M114.2 tok/s13.52 GB1L2 Cross-framework verified
NVIDIA RTX Sparkllama.cppQ4_K_M50.1 tok/s15.88 GB1L2 Cross-framework verified
NVIDIA RTX Sparkllama.cppQ5_K_S49.3 tok/s17.22 GB1L2 Cross-framework verified
NVIDIA RTX SparkOllamaQ3_K_S56.8 tok/s13.02 GB1L2 Cross-framework verified
NVIDIA RTX SparkOllamaQ4_K_M51.3 tok/s15.88 GB1L2 Cross-framework verified
NVIDIA RTX 4090llama.cppQ4_K_M147 tok/s10.64 GB2L2 Cross-framework verified
NVIDIA RTX 4090llama.cppQ4_K_S134.4 tok/s15.2 GB1L2 Cross-framework verified
NVIDIA RTX 4090StrataQ2_K254.9 tok/s11.34 GB1L2 Cross-framework verified
NVIDIA RTX 3090OllamaIQ3_S90.9 tok/s13.19 GB1L2 Cross-framework verified

91 of these records are not linked to a specific quantized build; the full set is on the search page.

Community discussion

No discussions yet — start the first one.

Start a discussion