Models › Llama 3.1 8B Instruct
Llama 3.1 8B Instruct: VRAM needs and speed by GPU
At Q4_K_M, Llama 3.1 8B Instruct needs about 5.9 GB of VRAM. We have 185 measured speed results for it.
8.03B parameters · license: llama3.1 · 131,072-token context · GGUF published Jul 23, 2024 · GGUF files on Hugging Face
Measured speeds
| Hardware | Generate tok/s | Prompt tok/s | Quant | Backend | Test | Source |
|---|---|---|---|---|---|---|
| NVIDIA H100 PCIe | 120 | 5,202 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA A100 80GB SXM | 110 | 3,571 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 3090 Ti | 110 | 4,024 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 4090 | 105 | 7,146 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX 5090 Laptop | 97.8 | 4,146 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX 3090 | 96.3 | 3,550 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA Tesla V100 16GB | 90.8 | 2,256 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 3080 Ti | 90.5 | 3,515 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX A6000 | 90.5 | 3,447 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX A5000 | 89.0 | 2,891 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 4080 | 88.3 | 4,792 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA Tesla V100 32GB | 88.2 | 2,177 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 6000 Ada | 86.1 | 6,148 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX PRO 6000 Blackwell | 85.8 | 6,346 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 3070 Ti | 83.2 | 2,509 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 3080 | 81.1 | 3,025 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA A40 | 80.6 | 3,270 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX A4500 | 77.9 | 2,498 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 4070 | 76.3 | 3,192 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA Quadro RTX 8000 | 74.0 | 1,677 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA TITAN V | 73.3 | 1,823 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 2080 Ti | 72.9 | 2,027 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M5 Max (40-core GPU) | 71.2 | 718 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA TITAN RTX | 71.2 | 2,226 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 4080 SUPER | 66.6 | 4,720 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX 5090 | 66.3 | 6,297 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 5070 Ti Laptop | 65.4 | 2,874 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA Quadro RTX 6000 | 64.9 | 1,891 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 4070 Ti | 64.2 | 3,769 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M3 Ultra (80-core GPU) | 63.0 | 1,086 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX 3070 Ti Laptop | 62.4 | 1,910 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M5 Max (32-core GPU) | 62.2 | 637 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 3060 Ti | 60.2 | 1,839 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 2080 SUPER | 59.8 | 1,693 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 3070 | 59.6 | 2,006 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M2 Ultra (60-core GPU) | 59.4 | 706 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 2080 | 58.4 | 1,546 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 5070 Laptop | 57.7 | 2,030 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 2070 SUPER | 57.6 | 1,444 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX A4000 | 56.5 | 1,931 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 5070 | 55.9 | 2,927 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 5060 Ti | 55.1 | 2,339 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| Apple M1 Ultra (64-core GPU) | 54.3 | 668 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 4070 Ti SUPER | 53.9 | 3,561 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 4000 Ada | 53.5 | 2,608 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 4070 SUPER | 53.4 | 3,336 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 3080 Laptop | 53.1 | 1,969 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| Apple M4 Max (40-core GPU) | 52.9 | 632 | Q4_K_M | llamafile | LocalScore avg · n=3 | LocalScore LocalScore LocalScore details |
| NVIDIA RTX 2070 | 52.4 | 1,252 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 5070 Ti | 50.1 | 3,116 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX 2060 SUPER | 49.9 | 1,205 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M1 Ultra (48-core GPU) | 48.9 | 534 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX PRO 6000 Blackwell Max-Q | 46.2 | 5,544 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA L40S | 45.9 | 5,328 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M2 Max (38-core GPU) | 45.5 | 479 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| Apple M4 Max (32-core GPU) | 45.5 | 540 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX A5000 Laptop | 45.5 | 1,497 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA L40 | 45.1 | 5,065 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M5 Pro (20-core GPU) | 44.7 | 411 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX 4000 SFF Ada | 44.4 | 2,004 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 3060 | 44.1 | 1,411 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| Apple M5 Pro (16-core GPU) | 43.5 | 338 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX 4060 Ti | 43.3 | 2,160 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA L4 | 42.9 | 2,056 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| AMD Radeon RX 6900 XT | 42.4 | 1,082 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA H100 SXM | 42.4 | 4,687 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| AMD Radeon RX 7900 XTX | 42.3 | 875 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX 3070 Laptop | 41.9 | 1,176 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 3500 Ada Laptop | 41.8 | 2,445 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 4070 Laptop | 40.9 | 2,131 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 3060 Laptop | 40.8 | 1,253 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX PRO 2000 Blackwell Laptop | 40.4 | 1,453 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA Quadro RTX 4000 | 40.0 | 992 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 4060 | 38.3 | 1,544 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 2060 | 38.1 | 960 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| AMD Radeon RX 7800 XT | 37.8 | 464 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| AMD Radeon RX 6800 XT | 37.6 | 951 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M3 Max (30-core GPU) | 37.6 | 450 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX 2000 Ada | 37.5 | 1,357 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA A100 80GB PCIe | 37.1 | 2,601 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA H100 NVL | 37.1 | 4,299 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA A10 | 36.5 | 2,085 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M1 Max (32-core GPU) | 36.2 | 369 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| Apple M2 Max (30-core GPU) | 36.2 | 362 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 4060 Laptop | 36.0 | 1,365 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX A2000 12GB | 35.7 | 976 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M3 Max (40-core GPU) | 35.6 | 480 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX 5080 | 35.5 | 4,045 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX 3050 | 34.5 | 965 | Q4_K_M | llamafile | LocalScore avg · n=3 | LocalScore LocalScore LocalScore details |
| NVIDIA DGX Spark (GB10) | 34.2 | 1,998 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| Apple M4 Pro (20-core GPU) | 32.7 | 360 | Q4_K_M | llamafile | LocalScore avg · n=3 | LocalScore LocalScore LocalScore details |
| NVIDIA Tesla P100 | 32.3 | 399 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| Apple M1 Max (24-core GPU) | 32.1 | 306 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 4050 Laptop | 31.4 | 1,201 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| AMD Radeon RX 6800 | 30.9 | 737 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M4 Pro (16-core GPU) | 30.2 | 297 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX 2000 Ada Laptop | 29.5 | 1,151 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA Quadro RTX 3000 | 29.5 | 606 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX A2000 | 28.7 | 758 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX A2000 8GB Laptop | 27.5 | 790 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M2 Pro (19-core GPU) | 26.3 | 262 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| AMD Radeon RX 5700 XT | 25.4 | 256 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| AMD Radeon RX 6650 XT | 25.2 | 536 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M6 (12-core GPU) | 25.2 | 263 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M2 Pro (16-core GPU) | 24.2 | 220 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX 3050 6GB Laptop | 24.2 | 790 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 1000 Ada Laptop | 24.2 | 955 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M1 Pro (16-core GPU) | 21.8 | 202 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| AMD Radeon RX 6600 XT | 21.5 | 604 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M3 Pro (18-core GPU) | 21.4 | 282 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| Apple M3 Pro (14-core GPU) | 21.3 | 221 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| Apple M5 (10-core GPU) | 20.2 | 189 | Q4_K_M | llamafile | LocalScore avg · n=3 | LocalScore LocalScore LocalScore details |
| NVIDIA GTX 1080 | 20.2 | 652 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M1 Pro (14-core GPU) | 20.1 | 175 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX PRO 1000 Blackwell Laptop | 19.9 | 906 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M4 (10-core GPU) | 16.8 | 166 | Q4_K_M | llamafile | LocalScore avg · n=3 | LocalScore LocalScore LocalScore details |
| AMD Radeon RX 6700 XT | 16.4 | 597 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple A18 Pro (5-core GPU) | 15.8 | 116 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M4 (8-core GPU) | 15.3 | 134 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M2 (8-core GPU) | 15.1 | 129 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| Apple M2 (10-core GPU) | 14.7 | 142 | Q4_K_M | llamafile | LocalScore avg · n=3 | LocalScore LocalScore LocalScore details |
| Apple M3 (8-core GPU) | 13.7 | 125 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M1 (8-core GPU) | 11.9 | 113 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| Apple M3 (10-core GPU) | 11.9 | 124 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| Apple M1 (7-core GPU) | 11.4 | 96 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| AMD Radeon RX 6600 | 10.2 | 419 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA GTX 1660 SUPER | 9.6 | 129 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA GTX 1660 | 9.0 | 129 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| AMD Radeon RX 6950 XT | 6.1 | 1,024 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| AMD Radeon RX 7600 | 5.5 | 317 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA T1000 8GB | 5.2 | 74 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 3050 Laptop | 4.8 | 280 | Q4_K_M | llamafile | LocalScore avg · part CPU | LocalScore details |
| AMD Radeon RX 7700S | 4.6 | 289 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 3050 Ti Laptop | 4.3 | 290 | Q4_K_M | llamafile | LocalScore avg · part CPU | LocalScore details |
| AMD Radeon 780M | 3.5 | 110 | Q4_K_M | llamafile | LocalScore avg · n=5 | LocalScore LocalScore LocalScore LocalScore LocalScore details |
| NVIDIA GTX 1650 Ti | 2.8 | 66 | Q4_K_M | llamafile | LocalScore avg · part CPU | LocalScore details |
| NVIDIA Quadro P620 | 1.3 | 45 | Q4_K_M | llamafile | LocalScore avg · part CPU | LocalScore details |
Memory needed per quantization
| Quant | File size | Needs (estimate) | Smallest card it fits |
|---|---|---|---|
| IQ2_M | 2.9 GB | 3.9 GB | 8 GB cards |
| Q2_K | 3.2 GB | 4.2 GB | 8 GB cards |
| IQ3_XS | 3.5 GB | 4.5 GB | 8 GB cards |
| Q3_K_S | 3.7 GB | 4.7 GB | 8 GB cards |
| Q2_K_L | 3.7 GB | 4.7 GB | 8 GB cards |
| IQ3_M | 3.8 GB | 4.8 GB | 8 GB cards |
| Q3_K_M | 4.0 GB | 5.0 GB | 8 GB cards |
| Q3_K_L | 4.3 GB | 5.3 GB | 8 GB cards |
| IQ4_XS | 4.4 GB | 5.4 GB | 8 GB cards |
| IQ4_NL | 4.7 GB | 5.7 GB | 8 GB cards |
| Q4_K_S | 4.7 GB | 5.7 GB | 8 GB cards |
| Q3_K_XL | 4.8 GB | 5.8 GB | 8 GB cards |
| Q4_K_M | 4.9 GB | 5.9 GB | 8 GB cards |
| Q4_K_L | 5.3 GB | 6.3 GB | 8 GB cards |
| Q5_K_S | 5.6 GB | 6.6 GB | 8 GB cards |
| Q5_K_M | 5.7 GB | 6.7 GB | 8 GB cards |
| Q5_K_L | 6.1 GB | 7.1 GB | 8 GB cards |
| Q6_K | 6.6 GB | 7.6 GB | 12 GB cards |
| Q6_K_L | 6.9 GB | 7.9 GB | 12 GB cards |
| Q8_0 | 8.5 GB | 9.5 GB | 12 GB cards |
Expected speed on popular hardware
| Hardware | Best quant that fits | Expected tok/s |
|---|---|---|
| NVIDIA RTX 3060 | Q8_0 | 36.1 estimate |
| NVIDIA RTX 4060 Ti | Q5_K_M | 42.7 estimate |
| NVIDIA RTX 3090 | Q8_0 | 76.8 estimate |
| NVIDIA RTX 4090 | Q8_0 | 85.1 estimate |
| NVIDIA RTX 5090 | Q8_0 | 134 estimate |
| AMD Radeon RX 7900 XTX | Q8_0 | 85.4 estimate |
| Apple M4 Pro (20-core GPU) (64 GB) | Q8_0 | 27.1 estimate |
| Apple M3 Max (40-core GPU) (64 GB) | Q8_0 | 39.6 estimate |
| AMD Ryzen AI Max+ 395 (Radeon 8060S) | Q8_0 | 24.3 estimate |
Too big for your card?
Run Llama 3.1 8B Instruct on a rented 24-80 GB GPU by the hour.
Rent a bigger GPU on RunPodAffiliate link: if you sign up, RigBench may earn a referral credit. It never affects rankings or numbers.