Models › Qwen2.5-Coder-32B-Instruct
Qwen2.5-Coder-32B-Instruct: VRAM needs and speed by GPU
At Q4_K_M, Qwen2.5-Coder-32B-Instruct needs about 21.5 GB of VRAM. No measured runs yet. Speeds below are estimates. Submit a run.
32.76B parameters · license: apache-2.0 · 32,768-token context · GGUF published Nov 12, 2024 · GGUF files on Hugging Face
Memory needed per quantization
| Quant | File size | Needs (estimate) | Smallest card it fits |
|---|---|---|---|
| Q2_K | 12.3 GB | 13.9 GB | 16 GB cards |
| Q3_K_M | 15.9 GB | 17.6 GB | 24 GB cards |
| Q4_K_M | 19.9 GB | 21.5 GB | 24 GB cards |
| Q5_K_M | 23.3 GB | 24.9 GB | 32 GB cards |
| Q6_K | 26.9 GB | 28.5 GB | 32 GB cards |
| Q8_0 | 34.8 GB | 36.4 GB | 48 GB cards |
Expected speed on popular hardware
| Hardware | Best quant that fits | Expected tok/s |
|---|---|---|
| NVIDIA RTX 3060 | Does not fit | |
| NVIDIA RTX 4060 Ti | Does not fit | |
| NVIDIA RTX 3090 | Q4_K_M | 33.0 estimate |
| NVIDIA RTX 4090 | Q4_K_M | 36.6 estimate |
| NVIDIA RTX 5090 | Q6_K | 42.7 estimate |
| AMD Radeon RX 7900 XTX | Q4_K_M | 36.8 estimate |
| Apple M4 Pro (20-core GPU) (64 GB) | Q8_0 | 6.6 estimate |
| Apple M3 Max (40-core GPU) (64 GB) | Q8_0 | 9.7 estimate |
| AMD Ryzen AI Max+ 395 (Radeon 8060S) | Q8_0 | 6.0 estimate |
Too big for your card?
Run Qwen2.5-Coder-32B-Instruct on a rented 24-80 GB GPU by the hour.
Rent a bigger GPU on RunPodAffiliate link: if you sign up, RigBench may earn a referral credit. It never affects rankings or numbers.