GPUs & Macs › NVIDIA P102-100
NVIDIA P102-100: local LLM speed
Llama 2 7B (Q4_0) generates 66.5 tokens/s on the NVIDIA P102-100 (Vulkan), and processes prompts at 575 tokens/s.
Measured results
| Model | Generate tok/s | Prompt tok/s | Quant | Backend | Test | Source |
|---|---|---|---|---|---|---|
| Llama 2 7B | 66.5 | 575 | Q4_0 | Vulkan FA on | tg128 | llama.cpp #10879 details |
| Llama 2 7B | 62.8 | 511 | Q4_0 | Vulkan FA off | tg128 | llama.cpp #10879 details |
What fits on the NVIDIA P102-100
Estimates for this chip are scaled from a measured run: Llama 2 7B Q4_0 at 66.5 tok/s (Vulkan, source). Fit assumes the whole model in GPU memory with ~8K context.
| Model | Best quant that fits | File | Expected tok/s |
|---|---|---|---|
| Qwen3.5-9B | UD-IQ2_XXS | 3.2 GB | 79.6 estimate |
| Qwen3.5-4B | Q6_K | 3.5 GB | 72.1 estimate |
| Qwen3-4B | Q6_K | 3.3 GB | 76.9 estimate |
| gemma-3-1b-it | Q8_0 | 1.1 GB | 212 estimate |
| gemma-4-E2B-it | Q4_0 | 2.8 GB | 89.4 estimate |
| DeepSeek-R1-0528-Qwen3-8B | UD-IQ3_XXS | 3.4 GB | 74.5 estimate |
| Llama-3.2-3B-Instruct | Q8_0 | 3.4 GB | 74.3 estimate |
| Phi-4-mini-instruct | Q6_K | 3.2 GB | 80.5 estimate |
| gemma-3-4b-it | Q6_K | 3.2 GB | 79.6 estimate |
| gemma-3-12b-it | UD-IQ1_M | 3.2 GB | 78.2 estimate |
| Qwen3.5-0.8B | Q8_0 | 0.8 GB | 212 estimate |
| Llama-3.2-3B-Instruct-uncensored | Q6_K | 3.0 GB | 85.6 estimate |
| Qwen3-0.6B | Q8_0 | 0.8 GB | 212 estimate |
| Qwen2.5-Coder-7B-Instruct-abliterated | Q3_K_S | 3.5 GB | 72.8 estimate |
Similar hardware
NVIDIA P106-100 · NVIDIA P104-100 · NVIDIA Tesla P4 · NVIDIA Tesla K80 · NVIDIA Tesla P100 · NVIDIA Tesla T4 · NVIDIA Tesla V100 16GB · NVIDIA A30