Models › Mistral-Small-3.2-24B-Instruct-2506

Mistral-Small-3.2-24B-Instruct-2506: VRAM needs and speed by GPU

At Q4_K_M, Mistral-Small-3.2-24B-Instruct-2506 needs about 15.7 GB of VRAM. No measured runs yet. Speeds below are estimates. Submit a run.

23.57B parameters · license: apache-2.0 · 131,072-token context · GGUF published Jun 20, 2025 · GGUF files on Hugging Face

Memory needed per quantization

QuantFile sizeNeeds (estimate)Smallest card it fits
UD-IQ1_S5.6 GB6.9 GB8 GB cards
UD-IQ1_M6.0 GB7.4 GB8 GB cards
UD-IQ2_XXS6.8 GB8.1 GB12 GB cards
UD-IQ2_M8.2 GB9.6 GB12 GB cards
Q2_K8.9 GB10.3 GB12 GB cards
Q2_K_L9.0 GB10.4 GB12 GB cards
UD-Q2_K_XL9.3 GB10.7 GB12 GB cards
UD-IQ3_XXS9.4 GB10.8 GB12 GB cards
Q3_K_S10.4 GB11.8 GB16 GB cards
Q3_K_M11.5 GB12.9 GB16 GB cards
UD-Q3_K_XL11.9 GB13.2 GB16 GB cards
IQ4_XS12.8 GB14.2 GB16 GB cards
IQ4_NL13.5 GB14.9 GB16 GB cards
Q4_013.5 GB14.9 GB16 GB cards
Q4_K_S13.5 GB14.9 GB16 GB cards
Q4_K_M14.3 GB15.7 GB24 GB cards
UD-Q4_K_XL14.5 GB15.9 GB24 GB cards
Q4_114.9 GB16.3 GB24 GB cards
Q5_K_S16.3 GB17.7 GB24 GB cards
Q5_K_M16.8 GB18.2 GB24 GB cards
UD-Q5_K_XL16.8 GB18.2 GB24 GB cards
Q6_K19.3 GB20.7 GB24 GB cards
UD-Q6_K_XL20.8 GB22.2 GB24 GB cards
Q8_025.1 GB26.4 GB32 GB cards
UD-Q8_K_XL29.0 GB30.4 GB32 GB cards
BF1647.2 GB48.5 GB96 GB cards

Expected speed on popular hardware

HardwareBest quant that fitsExpected tok/s
NVIDIA RTX 3060UD-IQ3_XXS32.7 estimate
NVIDIA RTX 4060 TiUD-IQ1_M40.7 estimate
NVIDIA RTX 3090Q6_K33.9 estimate
NVIDIA RTX 4090Q6_K37.6 estimate
NVIDIA RTX 5090Q8_045.8 estimate
AMD Radeon RX 7900 XTXQ6_K37.7 estimate
Apple M4 Pro (20-core GPU) (64 GB)Q8_09.2 estimate
Apple M3 Max (40-core GPU) (64 GB)Q8_013.5 estimate
AMD Ryzen AI Max+ 395 (Radeon 8060S)Q8_08.3 estimate

Check your own card

Too big for your card?

Run Mistral-Small-3.2-24B-Instruct-2506 on a rented 24-80 GB GPU by the hour.

Rent a bigger GPU on RunPod

Affiliate link: if you sign up, RigBench may earn a referral credit. It never affects rankings or numbers.