Models › Llama 3.2 1B Instruct
Llama 3.2 1B Instruct: VRAM needs and speed by GPU
At Q4_K_M, Llama 3.2 1B Instruct needs about 1.6 GB of VRAM. We have 230 measured speed results for it.
1.24B parameters · license: llama3.2 · GGUF files on Hugging Face
Measured speeds
| Hardware | Generate tok/s | Prompt tok/s | Quant | Backend | Test | Source |
|---|---|---|---|---|---|---|
| NVIDIA RTX 4090 D | 359 | 24,549 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 3090 Ti | 354 | 15,120 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA H100 PCIe | 335 | 18,590 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 3080 Ti | 313 | 14,143 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX A5000 | 310 | 12,138 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA A100 80GB SXM | 308 | 15,370 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX A6000 | 302 | 12,696 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX 3070 Ti | 297 | 10,074 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 4090 | 292 | 22,429 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX A4500 | 286 | 10,422 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA A40 | 284 | 12,156 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 4070 | 283 | 12,717 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 6000 Ada | 271 | 23,450 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX 3090 | 270 | 13,482 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA Tesla V100 32GB | 265 | 9,833 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M5 Max (40-core GPU) | 262 | 4,307 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA Tesla V100 16GB | 258 | 9,419 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA Quadro RTX 6000 | 253 | 9,138 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 3080 | 242 | 11,669 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX 4080 | 241 | 14,496 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA L40S | 238 | 21,532 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA Quadro RTX 8000 | 234 | 8,780 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA TITAN V | 232 | 7,853 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M5 Max (32-core GPU) | 229 | 3,635 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA Quadro RTX 5000 | 228 | 7,220 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 2080 Ti | 222 | 8,722 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA TITAN RTX | 217 | 9,273 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 3070 | 214 | 8,074 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 2080 | 210 | 6,747 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 4000 Ada | 210 | 10,448 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 4090 Laptop | 207 | 11,965 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX A5000 Laptop | 202 | 7,018 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 2060 SUPER | 199 | 5,616 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 4080 SUPER | 194 | 17,108 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX 3060 Ti | 193 | 7,644 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX PRO 6000 Blackwell | 191 | 20,440 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 2070 | 190 | 5,816 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 4000 SFF Ada | 189 | 8,753 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA L4 | 188 | 9,268 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 2080 SUPER | 186 | 7,096 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 3060 | 185 | 6,042 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 5060 | 185 | 7,818 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 4070 SUPER | 183 | 13,280 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 3500 Ada Laptop | 181 | 8,206 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M4 Max (40-core GPU) | 180 | 3,817 | Q4_K_M | llamafile | LocalScore avg · n=3 | LocalScore LocalScore LocalScore details |
| NVIDIA RTX A4000 | 180 | 7,751 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M3 Ultra (80-core GPU) | 178 | 5,300 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX 5070 Ti Laptop | 175 | 11,612 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M5 Pro (20-core GPU) | 175 | 2,409 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| Apple M2 Ultra (60-core GPU) | 174 | 3,290 | Q4_K_M | llamafile | LocalScore avg · n=3 | LocalScore LocalScore LocalScore details |
| Apple M3 Ultra (60-core GPU) | 173 | 4,394 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 5060 Ti | 170 | 8,814 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX 2070 SUPER | 170 | 5,537 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX 2000 Ada | 168 | 6,087 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 4060 Ti | 167 | 8,581 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| Apple M4 Max (32-core GPU) | 166 | 3,255 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 4070 Ti | 165 | 12,818 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| AMD Radeon RX 9070 XT | 162 | 2,708 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 4070 Ti SUPER | 161 | 13,268 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA TITAN Xp | 160 | 4,221 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 5090 Laptop | 160 | 11,638 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX 3080 Laptop | 157 | 7,533 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 3060 Laptop | 151 | 5,430 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M1 Ultra (64-core GPU) | 151 | 2,977 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA GTX 1080 Ti | 147 | 4,098 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M3 Max (40-core GPU) | 146 | 3,292 | Q4_K_M | llamafile | LocalScore avg · n=3 | LocalScore LocalScore LocalScore details |
| Apple M2 Max (38-core GPU) | 145 | 2,466 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX 5070 Ti | 145 | 12,339 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX 4070 Laptop | 143 | 7,398 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 5080 | 140 | 13,261 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX A2000 12GB | 140 | 4,549 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M1 Ultra (48-core GPU) | 138 | 2,582 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 3050 Laptop | 137 | 3,705 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 4060 Laptop | 137 | 6,340 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 2060 | 137 | 4,479 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 4060 | 134 | 5,352 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M3 Max (30-core GPU) | 133 | 2,537 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA DGX Spark (GB10) | 132 | 7,127 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX PRO 2000 Blackwell Laptop | 131 | 5,884 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 3050 Ti Laptop | 130 | 4,087 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M2 Max (30-core GPU) | 128 | 2,032 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| AMD Radeon RX 7900 GRE | 125 | 3,610 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 3070 Laptop | 124 | 4,930 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX A1000 Laptop | 123 | 3,665 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M1 Max (32-core GPU) | 121 | 1,982 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX 5090 | 120 | 16,760 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA Tesla P100 | 120 | 2,380 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| Apple M4 Pro (20-core GPU) | 119 | 2,131 | Q4_K_M | llamafile | LocalScore avg · n=3 | LocalScore LocalScore LocalScore details |
| NVIDIA RTX 4050 Laptop | 118 | 4,917 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA L40 | 118 | 17,686 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| AMD Radeon RX 7800 XT | 117 | 2,723 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX A2000 | 117 | 3,811 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA GTX 1070 Ti | 116 | 2,980 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M4 Pro (16-core GPU) | 111 | 1,807 | Q4_K_M | llamafile | LocalScore avg · n=3 | LocalScore LocalScore LocalScore details |
| NVIDIA RTX A2000 8GB Laptop | 109 | 3,399 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX PRO 6000 Blackwell Max-Q | 106 | 16,184 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX PRO 6000 Blackwell Server Edition | 106 | 18,081 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX A2000 Laptop | 105 | 3,084 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA Tesla P40 | 105 | 2,662 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 3050 | 104 | 3,347 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA GTX 1070 | 102 | 2,547 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 5070 | 101 | 8,328 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M2 Pro (19-core GPU) | 99.9 | 1,473 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| Apple M1 Max (24-core GPU) | 99.8 | 1,626 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA RTX 3050 6GB Laptop | 99.5 | 3,329 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| AMD Radeon RX 7900 XT | 99.2 | 4,957 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M5 (10-core GPU) | 99.1 | 1,275 | Q4_K_M | llamafile | LocalScore avg · n=3 | LocalScore LocalScore LocalScore details |
| NVIDIA H200 | 98.5 | 14,696 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA H100 SXM | 96.2 | 14,608 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA Quadro P4000 | 94.1 | 2,194 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA A10 | 93.5 | 8,777 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| AMD Radeon RX 6800 XT | 91.6 | 4,377 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M2 Pro (16-core GPU) | 91.2 | 1,304 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| Apple M3 Pro (18-core GPU) | 89.4 | 1,596 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| AMD Radeon RX 6650 XT | 88.2 | 2,641 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M3 Pro (14-core GPU) | 87.9 | 1,311 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA Quadro P6000 | 86.3 | 2,874 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA H100 NVL | 83.6 | 13,586 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA A100 80GB PCIe | 81.9 | 10,806 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 500 Ada Laptop | 81.6 | 3,624 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA GTX 1080 | 80.3 | 2,785 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| AMD Radeon RX 7900 XTX | 78.4 | 4,089 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| Apple M1 Pro (16-core GPU) | 77.8 | 1,164 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA GTX 1060 | 76.9 | 1,863 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M4 (10-core GPU) | 74.3 | 1,045 | Q4_K_M | llamafile | LocalScore avg · n=3 | LocalScore LocalScore LocalScore details |
| NVIDIA RTX A4500 Laptop | 72.5 | 3,868 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M1 Pro (14-core GPU) | 72.1 | 1,055 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA P106-100 | 70.3 | 1,758 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| AMD Radeon RX 6800 | 69.0 | 3,322 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M3 (10-core GPU) | 65.9 | 924 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| Apple M4 (8-core GPU) | 65.9 | 898 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| AMD Radeon RX 6700 XT | 64.3 | 2,883 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| AMD Radeon RX 6900 XT | 62.8 | 4,411 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA L40S-8Q | 62.4 | 12,563 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M3 (8-core GPU) | 60.3 | 804 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA Quadro P2000 | 58.3 | 1,353 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| AMD Radeon RX 7600 XT | 57.1 | 1,502 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| AMD Radeon Pro V620 | 57.0 | 3,905 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| AMD Radeon RX 5500 XT | 56.2 | 174 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M2 (10-core GPU) | 55.6 | 814 | Q4_K_M | llamafile | LocalScore avg · n=3 | LocalScore LocalScore LocalScore details |
| AMD Radeon RX 6600 | 52.9 | 2,029 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA RTX 2050 | 47.9 | 1,770 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA Jetson AGX Orin 64GB | 45.3 | 1,157 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA GTX 1050 Ti | 45.2 | 1,014 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA P106-090 | 39.2 | 923 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| Apple M1 (8-core GPU) | 38.9 | 579 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| Apple M1 (7-core GPU) | 38.0 | 529 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| Apple M2 (8-core GPU) | 37.8 | 553 | Q4_K_M | llamafile | LocalScore avg · n=2 | LocalScore LocalScore details |
| NVIDIA GTX 1660 Ti | 37.1 | 829 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
| NVIDIA GTX 1660 SUPER | 35.2 | 799 | Q4_K_M | llamafile | LocalScore avg | LocalScore details |
Memory needed per quantization
| Quant | File size | Needs (estimate) | Smallest card it fits |
|---|---|---|---|
| IQ3_M | 0.7 GB | 1.5 GB | 8 GB cards |
| Q3_K_L | 0.7 GB | 1.6 GB | 8 GB cards |
| IQ4_XS | 0.7 GB | 1.6 GB | 8 GB cards |
| Q4_0 | 0.8 GB | 1.6 GB | 8 GB cards |
| Q4_K_S | 0.8 GB | 1.6 GB | 8 GB cards |
| Q3_K_XL | 0.8 GB | 1.6 GB | 8 GB cards |
| Q4_K_M | 0.8 GB | 1.6 GB | 8 GB cards |
| Q4_K_L | 0.9 GB | 1.7 GB | 8 GB cards |
| Q5_K_S | 0.9 GB | 1.7 GB | 8 GB cards |
| Q5_K_M | 0.9 GB | 1.7 GB | 8 GB cards |
| Q5_K_L | 1.0 GB | 1.8 GB | 8 GB cards |
| Q6_K | 1.0 GB | 1.9 GB | 8 GB cards |
| Q6_K_L | 1.1 GB | 1.9 GB | 8 GB cards |
| Q8_0 | 1.3 GB | 2.2 GB | 8 GB cards |
| F16 | 2.5 GB | 3.3 GB | 8 GB cards |
Expected speed on popular hardware
| Hardware | Best quant that fits | Expected tok/s |
|---|---|---|
| NVIDIA RTX 3060 | Q8_0 | 233 estimate |
| NVIDIA RTX 4060 Ti | Q8_0 | 185 estimate |
| NVIDIA RTX 3090 | Q8_0 | 497 estimate |
| NVIDIA RTX 4090 | Q8_0 | 550 estimate |
| NVIDIA RTX 5090 | Q8_0 | 869 estimate |
| AMD Radeon RX 7900 XTX | Q8_0 | 552 estimate |
| Apple M4 Pro (20-core GPU) (64 GB) | Q8_0 | 175 estimate |
| Apple M3 Max (40-core GPU) (64 GB) | Q8_0 | 256 estimate |
| AMD Ryzen AI Max+ 395 (Radeon 8060S) | Q8_0 | 157 estimate |
Too big for your card?
Run Llama 3.2 1B Instruct on a rented 24-80 GB GPU by the hour.
Rent a bigger GPU on RunPodAffiliate link: if you sign up, RigBench may earn a referral credit. It never affects rankings or numbers.