Home › Models › Devstral Small 2 24B
Devstral Small 2 24B: VRAM needs and speed by GPU
At Q4_K_M, Devstral Small 2 24B needs about 15.7 GB of VRAM. We have 1 measured speed results for it.
23.57B parameters · license: other · GGUF files on Hugging Face
Measured speeds
| Chip | Generation tok/s | Prompt tok/s | Quant | Backend | Test | Source |
|---|---|---|---|---|---|---|
| Intel Arc Pro B70 | 30.0 | 987 | Q4_K_M | SYCL FA on | tg128 | PMZFX intel-arc-pro-b70-benchmarks (MIT) details |
Memory needed per quantization
| Quant | File size | Needs (estimate) | Smallest card it fits |
|---|---|---|---|
| UD-IQ1_S | 5.6 GB | 6.9 GB | 8 GB cards |
| UD-IQ1_M | 6.0 GB | 7.4 GB | 8 GB cards |
| UD-IQ2_XXS | 6.8 GB | 8.1 GB | 12 GB cards |
| UD-IQ2_M | 8.2 GB | 9.6 GB | 12 GB cards |
| Q2_K | 8.9 GB | 10.3 GB | 12 GB cards |
| Q2_K_L | 9.0 GB | 10.4 GB | 12 GB cards |
| UD-Q2_K_XL | 9.3 GB | 10.7 GB | 12 GB cards |
| UD-IQ3_XXS | 9.4 GB | 10.8 GB | 12 GB cards |
| Q3_K_S | 10.4 GB | 11.8 GB | 16 GB cards |
| Q3_K_M | 11.5 GB | 12.9 GB | 16 GB cards |
| UD-Q3_K_XL | 11.9 GB | 13.2 GB | 16 GB cards |
| IQ4_XS | 12.8 GB | 14.2 GB | 16 GB cards |
| IQ4_NL | 13.5 GB | 14.9 GB | 16 GB cards |
| Q4_0 | 13.5 GB | 14.9 GB | 16 GB cards |
| Q4_K_S | 13.5 GB | 14.9 GB | 16 GB cards |
| Q4_K_M | 14.3 GB | 15.7 GB | 24 GB cards |
| UD-Q4_K_XL | 14.5 GB | 15.9 GB | 24 GB cards |
| Q4_1 | 14.9 GB | 16.3 GB | 24 GB cards |
| Q5_K_S | 16.3 GB | 17.7 GB | 24 GB cards |
| Q5_K_M | 16.8 GB | 18.2 GB | 24 GB cards |
| UD-Q5_K_XL | 16.8 GB | 18.2 GB | 24 GB cards |
| Q6_K | 19.3 GB | 20.7 GB | 24 GB cards |
| UD-Q6_K_XL | 20.8 GB | 22.2 GB | 24 GB cards |
| Q8_0 | 25.1 GB | 26.4 GB | 32 GB cards |
| UD-Q8_K_XL | 29.0 GB | 30.4 GB | 32 GB cards |
| BF16 | 47.2 GB | 48.5 GB | 96 GB cards |
Expected speed on popular hardware
| Hardware | Best quant that fits | Expected tok/s |
|---|---|---|
| NVIDIA RTX 3060 | UD-IQ3_XXS | 32.7 estimate |
| NVIDIA RTX 4060 Ti | UD-IQ1_M | 40.7 estimate |
| NVIDIA RTX 3090 | Q6_K | 33.9 estimate |
| NVIDIA RTX 4090 | Q6_K | 45.0 estimate |
| NVIDIA RTX 5090 | Q8_0 | 45.8 estimate |
| AMD Radeon RX 7900 XTX | Q6_K | 37.7 estimate |
| Apple M4 Pro (20-core GPU) (64 GB) | Q8_0 | 9.2 estimate |
| Apple M3 Max (40-core GPU) (64 GB) | Q8_0 | 13.5 estimate |
| AMD Ryzen AI Max+ 395 (Radeon 8060S) | Q8_0 | 8.3 estimate |
Too big for your card?
Run Devstral Small 2 24B on a rented 24-80 GB GPU by the hour.
Rent a bigger GPU on RunPodAffiliate link: if you sign up, RigBench may earn a referral credit. It never affects rankings or numbers.