GPUs & Macs › NVIDIA TITAN V

NVIDIA TITAN V: local LLM speed

Llama 2 7B (Q4_0) generates 112 tokens/s on the NVIDIA TITAN V (CUDA), and processes prompts at 2,481 tokens/s.

Measured results

ModelGenerate
tok/s		Prompt
tok/s		QuantBackendTestSource
Llama 2 7B1122,481Q4_0CUDA FA ontg128llama.cpp #15013 details
Llama 2 7B109793Q4_0Vulkan FA ontg128llama.cpp #10879 details
Llama 2 7B1092,617Q4_0CUDA FA offtg128llama.cpp #15013 details
Llama 2 7B105796Q4_0Vulkan FA offtg128llama.cpp #10879 details
Llama 3.2 1B Instruct2327,853Q4_K_Mllamafile LocalScore avgLocalScore details
Llama 3.1 8B Instruct73.31,823Q4_K_Mllamafile LocalScore avgLocalScore details
Qwen2.5 14B Instruct42.2962Q4_K_Mllamafile LocalScore avgLocalScore details

What fits on the NVIDIA TITAN V

Estimates for this chip are scaled from a measured run: Llama 2 7B Q4_0 at 112 tok/s (CUDA, source). Fit assumes the whole model in GPU memory with ~8K context.

ModelBest quant that fitsFileExpected tok/s
Qwen3.8-27BUD-IQ2_XXS7.3 GB59.0 estimate
gemma-4-12B-itQ6_K9.8 GB43.8 estimate
Qwen3.5-9BQ8_09.5 GB45.0 estimate
Qwen3.5-4BQ8_04.5 GB95.7 estimate
gemma-4-26B-A4B-itUD-IQ2_M10.0 GB139 estimate
MiMo-V2.6-Distill-Qwen-9BQ8_09.5 GB44.9 estimate
gemma-4-E4B-itQ8_08.2 GB52.3 estimate
Qwen3-4BQ8_04.3 GB100 estimate
gemma-4-31B-itUD-IQ2_XXS8.5 GB50.2 estimate
gemma-4-12B-it-qat-q4_0-unquantizedUD-Q4_K_XL6.7 GB63.8 estimate
gemma-3-1b-itQ8_01.1 GB357 estimate
gemma-4-E2B-itQ8_05.0 GB86.3 estimate
DeepSeek-R1-0528-Qwen3-8BQ8_08.7 GB49.2 estimate
Llama-3.2-3B-InstructQ8_03.4 GB125 estimate

Open in the picker

Similar hardware

NVIDIA RTX 5070 · NVIDIA TITAN Xp · NVIDIA RTX 3060 · NVIDIA RTX 4070 Ti · NVIDIA RTX 4070 SUPER · NVIDIA RTX 4070 · NVIDIA RTX 3080 Ti · NVIDIA RTX 2080 Ti