Llama-3.2-3B-Instruct

3BChatby Meta Start a discussion

0

Quantized builds

106

Benchmark records

1

Discussions

Quantized builds

Community and official quantized releases; "linked records" counts only benchmarks tied to that exact release.

No quantized builds cataloged yet.

Benchmark data

50 configurations

HardwareFrameworkQuantDecodeVRAM usedSamplesEvidence
NVIDIA RTX 4090vLLMIQ3_S327.6 tok/s6.26 GB2L2 Cross-framework verified
NVIDIA RTX 4090OllamaQ4_K_S336.8 tok/s7.01 GB1L2 Cross-framework verified
NVIDIA RTX 4090OllamaQ5_K_M263.1 tok/s7.96 GB1L2 Cross-framework verified
NVIDIA RTX 3090llama.cppQ2_K287.4 tok/s5.57 GB2L2 Cross-framework verified
NVIDIA RTX 3090vLLMFP16114 tok/s13.76 GB1L2 Cross-framework verified
NVIDIA RTX 3090vLLMQ3_K_M250.5 tok/s6.38 GB1L2 Cross-framework verified
AMD Radeon RX 7900 XTXvLLMQ6_K175.9 tok/s8.4 GB1L2 Cross-framework verified
AMD Radeon RX 7900 XTXOllamaQ3_K_L237.9 tok/s6.64 GB1L2 Cross-framework verified
NVIDIA RTX 5090llama.cppQ4_K_S375.6 tok/s7.01 GB2L2 Cross-framework verified
NVIDIA RTX 5090llama.cppQ5_K_S373.6 tok/s7.77 GB1L2 Cross-framework verified
NVIDIA RTX 5090vLLMIQ4_XS321.2 tok/s6.76 GB1L2 Cross-framework verified
NVIDIA RTX 5090vLLMQ6_K292.4 tok/s8.4 GB1L2 Cross-framework verified
NVIDIA RTX 5080llama.cppQ5_K_S381.8 tok/s7.77 GB1L2 Cross-framework verified
NVIDIA RTX 5080OllamaQ5_K_S312.2 tok/s7.77 GB1L2 Cross-framework verified
NVIDIA RTX 5080OllamaQ8_0294 tok/s9.35 GB1L2 Cross-framework verified
NVIDIA RTX 3060 12GBvLLMQ4_K_S182.9 tok/s7.01 GB1L2 Cross-framework verified
NVIDIA RTX 3060 12GBvLLMQ5_K_S145 tok/s7.77 GB1L2 Cross-framework verified
NVIDIA RTX 3060 12GBStrataQ6_K168.1 tok/s8.4 GB1L2 Cross-framework verified
NVIDIA RTX 4080 Superllama.cppIQ2_XS373 tok/s5.12 GB1L2 Cross-framework verified
NVIDIA RTX 4080 Superllama.cppIQ4_XS290 tok/s6.76 GB1L2 Cross-framework verified
NVIDIA RTX 4080 SuperStrataFP16277.1 tok/s13.76 GB1L2 Cross-framework verified
AMD Ryzen AI Max+ 395llama.cppQ8_061.6 tok/s9.35 GB1L2 Cross-framework verified
AMD Ryzen AI Max+ 395vLLMQ4_K_M89.3 tok/s7.27 GB1L2 Cross-framework verified
Apple M5 Maxllama.cppQ4_K_S188.3 tok/s7.01 GB1L2 Cross-framework verified
Apple M5 MaxOllamaIQ3_XS209.8 tok/s6.01 GB1L2 Cross-framework verified
Apple M5 Ultrallama.cppQ3_K_S338.3 tok/s6.2 GB1L2 Cross-framework verified
Apple M5 UltraOllamaQ3_K_S318.4 tok/s6.2 GB1L2 Cross-framework verified
NVIDIA H100 80GB HBM3llama.cppFP16379.9 tok/s13.76 GB1L2 Cross-framework verified
NVIDIA H100 80GB HBM3llama.cppQ5_K_M313.8 tok/s7.96 GB1L2 Cross-framework verified
NVIDIA H100 80GB HBM3OllamaFP16308.8 tok/s13.76 GB1L2 Cross-framework verified
NVIDIA H100 80GB HBM3OllamaIQ2_XS295.4 tok/s5.12 GB1L2 Cross-framework verified
NVIDIA H20llama.cppQ3_K_S171.3 tok/s6.2 GB1L2 Cross-framework verified
NVIDIA H20vLLMFP1676.2 tok/s13.76 GB1L2 Cross-framework verified
NVIDIA H20vLLMIQ3_XS177.1 tok/s6.01 GB1L2 Cross-framework verified
NVIDIA RTX PRO 6000 Blackwell 96GBllama.cppIQ3_XS338 tok/s6.01 GB1L2 Cross-framework verified
NVIDIA RTX PRO 6000 Blackwell 96GBllama.cppQ8_0331.1 tok/s9.35 GB1L2 Cross-framework verified
NVIDIA RTX PRO 6000 Blackwell 96GBOllamaIQ3_XS283.4 tok/s6.01 GB1L2 Cross-framework verified
NVIDIA RTX PRO 6000 Blackwell 96GBOllamaQ3_K_S364.1 tok/s6.2 GB1L2 Cross-framework verified
Intel Arc B580llama.cppQ2_K169.6 tok/s5.57 GB1L2 Cross-framework verified
Intel Arc B580llama.cppQ5_K_S120.7 tok/s7.77 GB1L2 Cross-framework verified
Intel Arc B580OllamaFP1662.5 tok/s11.4 GB1L2 Cross-framework verified
Apple M2 Ultrallama.cppQ5_K_M233.8 tok/s7.96 GB1L2 Cross-framework verified
Apple M2 UltraOllamaQ5_K_M207 tok/s7.96 GB1L2 Cross-framework verified
NVIDIA RTX Sparkllama.cppIQ3_S108.8 tok/s6.26 GB1L2 Cross-framework verified
NVIDIA RTX Sparkllama.cppQ4_K_M89.5 tok/s7.27 GB1L2 Cross-framework verified
NVIDIA RTX SparkOllamaIQ2_XS115.7 tok/s5.12 GB1L2 Cross-framework verified
NVIDIA RTX SparkOllamaQ4_K_M76.6 tok/s7.27 GB1L2 Cross-framework verified
NVIDIA RTX 4090StrataIQ1_S284.1 tok/s4.75 GB1L2 Cross-framework verified
NVIDIA RTX 4090StrataIQ2_XS281.2 tok/s5.12 GB1L2 Cross-framework verified
NVIDIA RTX 3090StrataFP16145.3 tok/s13.76 GB1L2 Cross-framework verified

106 of these records are not linked to a specific quantized build; the full set is on the search page.

Community discussion

View all discussions