Models › Qwen2.5-Coder-7B-Instruct-abliterated

Qwen2.5-Coder-7B-Instruct-abliterated: VRAM needs and speed by GPU

At Q4_K_M, Qwen2.5-Coder-7B-Instruct-abliterated needs about 5.7 GB of VRAM. No measured runs yet. Speeds below are estimates. Submit a run.

7.62B parameters · license: apache-2.0 · 32,768-token context · GGUF published Nov 13, 2024 · GGUF files on Hugging Face

Memory needed per quantization

QuantFile sizeNeeds (estimate)Smallest card it fits
IQ2_M2.8 GB3.8 GB8 GB cards
Q2_K3.0 GB4.0 GB8 GB cards
IQ3_XS3.3 GB4.3 GB8 GB cards
Q3_K_S3.5 GB4.5 GB8 GB cards
Q2_K_L3.5 GB4.5 GB8 GB cards
IQ3_M3.6 GB4.6 GB8 GB cards
Q3_K_M3.8 GB4.8 GB8 GB cards
Q3_K_L4.1 GB5.1 GB8 GB cards
IQ4_XS4.2 GB5.2 GB8 GB cards
Q4_04.4 GB5.4 GB8 GB cards
Q4_K_S4.5 GB5.4 GB8 GB cards
Q3_K_XL4.6 GB5.6 GB8 GB cards
Q4_K_M4.7 GB5.7 GB8 GB cards
Q4_K_L5.1 GB6.1 GB8 GB cards
Q5_K_S5.3 GB6.3 GB8 GB cards
Q5_K_M5.4 GB6.4 GB8 GB cards
Q5_K_L5.8 GB6.8 GB8 GB cards
Q6_K6.3 GB7.2 GB8 GB cards
Q6_K_L6.5 GB7.5 GB12 GB cards
Q8_08.1 GB9.1 GB12 GB cards
F1615.2 GB16.2 GB24 GB cards

Expected speed on popular hardware

HardwareBest quant that fitsExpected tok/s
NVIDIA RTX 3060Q8_038.0 estimate
NVIDIA RTX 4060 TiQ6_K39.1 estimate
NVIDIA RTX 3090Q8_081.0 estimate
NVIDIA RTX 4090Q8_089.7 estimate
NVIDIA RTX 5090Q8_0142 estimate
AMD Radeon RX 7900 XTXQ8_090.1 estimate
Apple M4 Pro (20-core GPU) (64 GB)Q8_028.6 estimate
Apple M3 Max (40-core GPU) (64 GB)Q8_041.7 estimate
AMD Ryzen AI Max+ 395 (Radeon 8060S)Q8_025.7 estimate

Check your own card

Too big for your card?

Run Qwen2.5-Coder-7B-Instruct-abliterated on a rented 24-80 GB GPU by the hour.

Rent a bigger GPU on RunPod

Affiliate link: if you sign up, RigBench may earn a referral credit. It never affects rankings or numbers.