Models › Qwen2.5 14B Instruct
Qwen2.5 14B Instruct: VRAM needs and speed by GPU
At Q4_K_M, Qwen2.5 14B Instruct needs about 10.2 GB of VRAM. We have 169 measured speed results for it.
14.77B parameters · license: apache-2.0 · GGUF files on Hugging Face
Measured speeds
| Hardware | Generate tok/s | Prompt tok/s | Quant | Backend | Test | Source |
|---|---|---|---|---|---|---|
| NVIDIA H100 PCIe | 72.2 | 2,973 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 4090 D | 70.5 | 3,905 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA A100 80GB SXM | 66.3 | 2,069 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 3090 Ti | 64.2 | 2,202 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX PRO 6000 Blackwell | 64.0 | 4,416 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 3090 | 56.2 | 1,950 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA Tesla V100 16GB | 53.0 | 1,184 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX A6000 | 52.0 | 1,883 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX A5000 | 50.4 | 1,532 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 4090 | 49.1 | 3,506 | Q4_K_M | llamafile | LocalScore avg · n=3 | LocalScore LocalScore LocalScore details |
| NVIDIA RTX 6000 Ada | 49.1 | 3,244 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA A40 | 46.2 | 1,825 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 3080 Ti | 46.1 | 1,793 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 4080 SUPER | 44.9 | 2,742 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX A4500 | 44.0 | 1,331 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA TITAN V | 42.2 | 962 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 3080 | 42.0 | 1,498 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX 5090 | 41.8 | 3,734 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA TITAN RTX | 41.5 | 1,215 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX 5090 Laptop | 40.5 | 1,550 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 4070 Ti | 40.2 | 2,176 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA Quadro RTX 8000 | 39.8 | 1,132 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA A40-48Q | 39.5 | 1,707 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 2080 Ti | 39.5 | 1,140 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M5 Max (40-core GPU) | 39.5 | 392 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA Tesla V100 32GB | 39.0 | 1,042 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX 4070 | 37.9 | 1,692 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA Quadro RTX 6000 | 37.1 | 1,077 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 4070 Ti SUPER | 36.8 | 2,147 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M2 Ultra (76-core GPU) | 36.6 | 470 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX 5070 Ti Laptop | 36.3 | 1,474 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M3 Ultra (80-core GPU) | 36.3 | 574 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX 4080 | 35.2 | 2,339 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M3 Ultra (60-core GPU) | 34.4 | 445 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M5 Max (32-core GPU) | 34.1 | 343 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M2 Ultra (60-core GPU) | 34.1 | 381 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX 5070 | 33.2 | 1,511 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| Apple M1 Ultra (64-core GPU) | 32.4 | 372 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX A4000 | 32.0 | 1,042 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 5070 Ti | 31.3 | 1,994 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX 4000 Ada | 30.0 | 1,417 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 5080 | 29.3 | 2,256 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| Apple M4 Max (40-core GPU) | 28.9 | 328 | Q4_K_M | llamafile | LocalScore avg · n=3 | LocalScore LocalScore LocalScore details |
| NVIDIA RTX 3080 Laptop | 28.2 | 858 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M1 Ultra (48-core GPU) | 27.8 | 290 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 5060 Ti | 27.7 | 1,306 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX A5000 Laptop | 27.6 | 839 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| AMD Radeon RX 7900 XTX | 26.8 | 490 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX 3060 | 26.6 | 759 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M4 Max (32-core GPU) | 25.6 | 285 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 3500 Ada Laptop | 25.1 | 1,298 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 4070 SUPER | 24.9 | 1,701 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA L40 | 24.7 | 2,713 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 4000 SFF Ada | 24.7 | 1,051 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX PRO 4500 Blackwell | 24.2 | 1,824 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M5 Pro (20-core GPU) | 24.0 | 217 | Q4_K_M | llamafile | LocalScore avg · n=3 | LocalScore LocalScore LocalScore details |
| NVIDIA RTX PRO 6000 Blackwell Max-Q | 23.7 | 2,623 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M3 Max (40-core GPU) | 23.6 | 287 | Q4_K_M | llamafile | LocalScore avg · n=3 | LocalScore LocalScore LocalScore details |
| NVIDIA L4 | 23.6 | 1,095 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA H100 SXM | 23.4 | 2,609 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX PRO 2000 Blackwell | 23.2 | 934 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 2060 | 22.8 | 595 | Q4_K_M | llamafile | LocalScore avg · part CPU | LocalScore details |
| AMD Radeon RX 7800 XT | 22.8 | 287 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| Apple M5 Pro (16-core GPU) | 22.1 | 173 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| Apple M2 Max (38-core GPU) | 22.0 | 226 | Q4_K_M | llamafile | LocalScore avg · n=3 | LocalScore LocalScore LocalScore details |
| AMD Radeon RX 6800 XT | 21.4 | 512 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA TITAN Xp | 21.1 | 412 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA A100 80GB PCIe | 20.8 | 1,438 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 2000 Ada | 20.7 | 740 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA H100 NVL | 20.5 | 2,354 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA GTX 1080 Ti | 20.2 | 419 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M3 Max (30-core GPU) | 20.1 | 227 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA A10 | 19.9 | 1,133 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA Tesla P100 | 19.8 | 193 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| Apple M1 Max (32-core GPU) | 19.8 | 194 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX A2000 12GB | 19.5 | 506 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA L40S | 19.3 | 2,868 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| AMD Radeon RX 6750 XT | 18.9 | 351 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| AMD Radeon RX 6900 XT | 18.5 | 569 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA DGX Spark (GB10) | 18.3 | 1,094 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| Apple M4 Pro (20-core GPU) | 18.0 | 186 | Q4_K_M | llamafile | LocalScore avg · n=3 | LocalScore LocalScore LocalScore details |
| NVIDIA RTX PRO 6000 Blackwell Server Edition | 17.9 | 2,581 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M1 Max (24-core GPU) | 16.7 | 147 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX 5000 Ada Laptop | 16.4 | 952 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M4 Pro (16-core GPU) | 16.1 | 151 | Q4_K_M | llamafile | LocalScore avg · n=3 | LocalScore LocalScore LocalScore details |
| AMD Radeon RX 6700 XT | 15.7 | 316 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA Tesla P40 | 15.4 | 329 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA Quadro P5000 | 14.9 | 303 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| AMD Radeon RX 6800 | 14.8 | 399 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 4060 Ti | 14.6 | 783 | Q4_K_M | llamafile | LocalScore avg · n=2 · part CPU | LocalScore LocalScore details |
| Apple M2 Max (30-core GPU) | 14.5 | 149 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M2 Pro (19-core GPU) | 14.1 | 137 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M2 Pro (16-core GPU) | 13.3 | 118 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA Tesla T4 | 13.3 | 450 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| AMD Instinct MI50 | 12.9 | 48 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M3 Pro (18-core GPU) | 11.9 | 146 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| Apple M3 Pro (14-core GPU) | 11.9 | 117 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| Apple M1 Pro (16-core GPU) | 11.8 | 105 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| Apple M1 Pro (14-core GPU) | 10.4 | 89 | Q4_K_M | llamafile | LocalScore avg · n=3 | LocalScore LocalScore LocalScore details |
| Apple M5 (10-core GPU) | 10.2 | 96 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| Apple M4 (10-core GPU) | 9.0 | 87 | Q4_K_M | llamafile | LocalScore avg · n=3 | LocalScore LocalScore LocalScore details |
| NVIDIA RTX A4500 Laptop | 7.9 | 302 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M2 (10-core GPU) | 7.5 | 71 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX 3060 Ti | 7.5 | 430 | Q4_K_M | llamafile | LocalScore avg · part CPU | LocalScore details |
| Apple M4 (8-core GPU) | 7.3 | 61 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M2 (8-core GPU) | 7.0 | 60 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M3 (10-core GPU) | 6.1 | 66 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 3070 | 5.6 | 398 | Q4_K_M | llamafile | LocalScore avg · part CPU | LocalScore details |
| NVIDIA A2 | 5.6 | 160 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M1 (8-core GPU) | 5.4 | 53 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 2080 | 5.3 | 324 | Q4_K_M | llamafile | LocalScore avg · part CPU | LocalScore details |
| Apple M1 (7-core GPU) | 4.8 | 41 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 2080 SUPER | 4.5 | 294 | Q4_K_M | llamafile | LocalScore avg · part CPU | LocalScore details |
| AMD Radeon AI PRO R9700 | 4.3 | 381 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 4070 Laptop | 3.8 | 289 | Q4_K_M | llamafile | LocalScore avg · part CPU | LocalScore details |
| NVIDIA L40S-8Q | 3.5 | 343 | Q4_K_M | llamafile | LocalScore avg · part CPU | LocalScore details |
| AMD Radeon RX 6950 XT | 3.3 | 549 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| AMD Radeon 780M | 2.5 | 78 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX PRO 2000 Blackwell Laptop | 2.1 | 230 | Q4_K_M | llamafile | LocalScore avg · part CPU | LocalScore details |
| NVIDIA GTX 1050 Ti | 1.2 | 37 | Q4_K_M | llamafile | LocalScore avg · part CPU | LocalScore details |
| NVIDIA RTX 3060 Laptop | 1.1 | 83 | Q4_K_M | llamafile | LocalScore avg · part CPU | LocalScore details |
Memory needed per quantization
| Quant | File size | Needs (estimate) | Smallest card it fits |
|---|---|---|---|
| IQ2_M | 5.4 GB | 6.5 GB | 8 GB cards |
| Q2_K | 5.8 GB | 6.9 GB | 8 GB cards |
| IQ3_XS | 6.4 GB | 7.6 GB | 12 GB cards |
| Q2_K_L | 6.5 GB | 7.7 GB | 12 GB cards |
| Q3_K_S | 6.7 GB | 7.8 GB | 12 GB cards |
| IQ3_M | 6.9 GB | 8.1 GB | 12 GB cards |
| Q3_K_M | 7.3 GB | 8.5 GB | 12 GB cards |
| Q3_K_L | 7.9 GB | 9.1 GB | 12 GB cards |
| IQ4_XS | 8.1 GB | 9.3 GB | 12 GB cards |
| Q4_0 | 8.5 GB | 9.7 GB | 12 GB cards |
| Q4_K_S | 8.6 GB | 9.7 GB | 12 GB cards |
| Q3_K_XL | 8.6 GB | 9.8 GB | 12 GB cards |
| Q4_K_M | 9.0 GB | 10.2 GB | 12 GB cards |
| Q4_K_L | 9.6 GB | 10.7 GB | 12 GB cards |
| Q5_K_S | 10.3 GB | 11.4 GB | 12 GB cards |
| Q5_K_M | 10.5 GB | 11.7 GB | 16 GB cards |
| Q5_K_L | 11.0 GB | 12.2 GB | 16 GB cards |
| Q6_K | 12.1 GB | 13.3 GB | 16 GB cards |
| Q6_K_L | 12.5 GB | 13.7 GB | 16 GB cards |
| Q8_0 | 15.7 GB | 16.9 GB | 24 GB cards |
| F16 | 29.5 GB | 30.7 GB | 32 GB cards |
Expected speed on popular hardware
| Hardware | Best quant that fits | Expected tok/s |
|---|---|---|
| NVIDIA RTX 3060 | Q5_K_S | 30.0 estimate |
| NVIDIA RTX 4060 Ti | Q2_K | 42.4 estimate |
| NVIDIA RTX 3090 | Q8_0 | 41.8 estimate |
| NVIDIA RTX 4090 | Q8_0 | 46.3 estimate |
| NVIDIA RTX 5090 | Q8_0 | 73.1 estimate |
| AMD Radeon RX 7900 XTX | Q8_0 | 46.5 estimate |
| Apple M4 Pro (20-core GPU) (64 GB) | Q8_0 | 14.7 estimate |
| Apple M3 Max (40-core GPU) (64 GB) | Q8_0 | 21.5 estimate |
| AMD Ryzen AI Max+ 395 (Radeon 8060S) | Q8_0 | 13.2 estimate |
Too big for your card?
Run Qwen2.5 14B Instruct on a rented 24-80 GB GPU by the hour.
Rent a bigger GPU on RunPodAffiliate link: if you sign up, RigBench may earn a referral credit. It never affects rankings or numbers.