Models › Llama 2 7B
Llama 2 7B: VRAM needs and speed by GPU
At Q4_K_M, Llama 2 7B needs about 5.1 GB of VRAM. We have 491 measured speed results for it.
6.74B parameters · license: llama2 · GGUF files on Hugging Face
Measured speeds
Memory needed per quantization
| Quant | File size | Needs (estimate) | Smallest card it fits |
|---|---|---|---|
| Q2_K | 2.8 GB | 3.8 GB | 8 GB cards |
| Q3_K_S | 2.9 GB | 3.9 GB | 8 GB cards |
| Q3_K_M | 3.3 GB | 4.3 GB | 8 GB cards |
| Q3_K_L | 3.6 GB | 4.6 GB | 8 GB cards |
| Q4_0 | 3.8 GB | 4.8 GB | 8 GB cards |
| Q4_K_S | 3.9 GB | 4.8 GB | 8 GB cards |
| Q4_K_M | 4.1 GB | 5.1 GB | 8 GB cards |
| Q5_0 | 4.7 GB | 5.6 GB | 8 GB cards |
| Q5_K_S | 4.7 GB | 5.6 GB | 8 GB cards |
| Q5_K_M | 4.8 GB | 5.8 GB | 8 GB cards |
| Q6_K | 5.5 GB | 6.5 GB | 8 GB cards |
| Q8_0 | 7.2 GB | 8.1 GB | 12 GB cards |
| F16 | 13.5 GB | 14.4 GB | 16 GB cards |
Expected speed on popular hardware
| Hardware | Best quant that fits | Expected tok/s |
|---|---|---|
| NVIDIA RTX 3060 | Q8_0 | 43.0 estimate |
| NVIDIA RTX 4060 Ti | Q6_K | 44.3 estimate |
| NVIDIA RTX 3090 | Q8_0 | 91.6 estimate |
| NVIDIA RTX 4090 | Q8_0 | 101 estimate |
| NVIDIA RTX 5090 | Q8_0 | 160 estimate |
| AMD Radeon RX 7900 XTX | Q8_0 | 102 estimate |
| Apple M4 Pro (20-core GPU) (64 GB) | Q8_0 | 32.3 estimate |
| Apple M3 Max (40-core GPU) (64 GB) | Q8_0 | 47.2 estimate |
| AMD Ryzen AI Max+ 395 (Radeon 8060S) | Q8_0 | 29.0 estimate |
Too big for your card?
Run Llama 2 7B on a rented 24-80 GB GPU by the hour.
Rent a bigger GPU on RunPodAffiliate link: if you sign up, RigBench may earn a referral credit. It never affects rankings or numbers.