Models › Mellum2.1-12B-A2.5B-Thinking

Mellum2.1-12B-A2.5B-Thinking: VRAM needs and speed by GPU

At Q4_K_M, Mellum2.1-12B-A2.5B-Thinking needs about 9.9 GB of VRAM. No measured runs yet. Speeds below are estimates. Submit a run.

12.15B parameters (mixture-of-experts, ~2.5B active) · license: apache-2.0 · 131,072-token context · GGUF published Oct 8, 2026 · GGUF files on Hugging Face

Memory needed per quantization

QuantFile sizeNeeds (estimate)Smallest card it fits
IQ2_M5.4 GB6.5 GB8 GB cards
Q2_K5.5 GB6.6 GB8 GB cards
Q3_K_S6.1 GB7.2 GB8 GB cards
IQ3_XXS6.3 GB7.4 GB8 GB cards
IQ3_XS6.3 GB7.4 GB8 GB cards
Q3_K_M6.7 GB7.8 GB12 GB cards
IQ4_XS7.0 GB8.1 GB12 GB cards
Q4_07.0 GB8.1 GB12 GB cards
Q3_K_L7.1 GB8.2 GB12 GB cards
IQ3_M7.2 GB8.3 GB12 GB cards
IQ4_NL7.7 GB8.8 GB12 GB cards
Q4_17.7 GB8.8 GB12 GB cards
Q4_K_S7.9 GB9.0 GB12 GB cards
Q4_K_M8.8 GB9.9 GB12 GB cards
Q5_K_S9.0 GB10.1 GB12 GB cards
Q4_K_L9.2 GB10.3 GB12 GB cards
Q5_K_M10.0 GB11.1 GB12 GB cards
Q6_K_S11.1 GB12.2 GB16 GB cards
Q6_K11.5 GB12.6 GB16 GB cards
Q6_K_L11.8 GB12.9 GB16 GB cards
Q8_012.9 GB14.0 GB16 GB cards
BF1624.3 GB25.4 GB32 GB cards

Expected speed on popular hardware

HardwareBest quant that fitsExpected tok/s
NVIDIA RTX 3060Q5_K_M74.2 estimate
NVIDIA RTX 4060 TiIQ3_XXS93.5 estimate
NVIDIA RTX 3090Q8_0122 estimate
NVIDIA RTX 4090Q8_0135 estimate
NVIDIA RTX 5090Q8_0213 estimate
AMD Radeon RX 7900 XTXQ8_0136 estimate
Apple M4 Pro (20-core GPU) (64 GB)Q8_043.0 estimate
Apple M3 Max (40-core GPU) (64 GB)Q8_062.8 estimate
AMD Ryzen AI Max+ 395 (Radeon 8060S)Q8_038.6 estimate

Check your own card

Too big for your card?

Run Mellum2.1-12B-A2.5B-Thinking on a rented 24-80 GB GPU by the hour.

Rent a bigger GPU on RunPod

Affiliate link: if you sign up, RigBench may earn a referral credit. It never affects rankings or numbers.