GPUs & Macs › NVIDIA Tesla V100 32GB

NVIDIA Tesla V100 32GB: local LLM speed

Llama 2 7B (Q4_0) generates 142 tokens/s on the NVIDIA Tesla V100 32GB (Vulkan), and processes prompts at 1,411 tokens/s.

Measured results

ModelGenerate
tok/s		Prompt
tok/s		QuantBackendTestSource
Llama 2 7B1421,411Q4_0Vulkan FA ontg128llama.cpp #10879 details
Llama 2 7B1352,974Q4_0CUDA FA ontg128llama.cpp #15013 details
Llama 2 7B1301,391Q4_0Vulkan FA offtg128llama.cpp #10879 details
Llama 2 7B1293,043Q4_0CUDA FA offtg128llama.cpp #15013 details
Llama 3.2 1B Instruct2659,833Q4_K_Mllamafile LocalScore avgLocalScore details
Llama 3.1 8B Instruct88.22,177Q4_K_Mllamafile LocalScore avgLocalScore details
Qwen2.5 14B Instruct39.01,042Q4_K_Mllamafile LocalScore avg · n=2LocalScore LocalScore details

What fits on the NVIDIA Tesla V100 32GB

Estimates for this chip are scaled from a measured run: Llama 2 7B Q4_0 at 142 tok/s (Vulkan, source). Fit assumes the whole model in GPU memory with ~8K context.

ModelBest quant that fitsFileExpected tok/s
Qwen3.8-27BQ8_029.0 GB18.7 estimate
gemma-4-12B-itQ8_012.7 GB42.9 estimate
Qwen3.5-9BQ8_09.5 GB57.0 estimate
Qwen3.5-4BQ8_04.5 GB121 estimate
Qwen3.6-35B-A3BUD-Q6_K29.3 GB108 estimate
gemma-4-26B-A4B-itQ8_026.9 GB65.8 estimate
MiMo-V2.6-Distill-Qwen-9BQ8_09.5 GB56.9 estimate
gemma-4-E4B-itQ8_08.2 GB66.3 estimate
Muse-Glimmer-30BQ8_029.6 GB18.3 estimate
Qwen3-4BQ8_04.3 GB127 estimate
gemma-4-31B-itQ6_K25.2 GB21.6 estimate
gemma-4-26B-A4B-it-qat-q4_0-unquantizedUD-Q4_K_XL14.2 GB124 estimate
gemma-4-12B-it-qat-q4_0-unquantizedUD-Q4_K_XL6.7 GB80.9 estimate
gemma-3-1b-itQ8_01.1 GB453 estimate

Open in the picker

Similar hardware

NVIDIA A30 · NVIDIA Tesla P40 · NVIDIA Tesla M40 · NVIDIA L4 · NVIDIA A10 · NVIDIA L40 · NVIDIA A40 · NVIDIA Tesla P100