GPUs & Macs › NVIDIA RTX 4080 SUPER

NVIDIA RTX 4080 SUPER: local LLM speed

Llama 2 7B (Q4_0) generates 150 tokens/s on the NVIDIA RTX 4080 SUPER (Vulkan), and processes prompts at 8,007 tokens/s.

Measured results

ModelGenerate
tok/s		Prompt
tok/s		QuantBackendTestSource
Llama 2 7B1508,007Q4_0Vulkan FA ontg128llama.cpp #10879 details
Llama 2 7B1488,125Q4_0CUDA FA offtg128llama.cpp #15013 details
Llama 2 7B1479,439Q4_0CUDA FA ontg128llama.cpp #15013 details
Llama 2 7B1477,101Q4_0Vulkan FA offtg128llama.cpp #10879 details
Llama 3.2 1B Instruct19417,108Q4_K_Mllamafile LocalScore avg · n=2LocalScore LocalScore details
gpt-oss 20B1878,171MXFP4CUDA FA ontg128llama.cpp #15396 details
Llama 3.1 8B Instruct66.64,720Q4_K_Mllamafile LocalScore avg · n=2LocalScore LocalScore details
Qwen2.5 14B Instruct44.92,742Q4_K_Mllamafile LocalScore avg · n=2LocalScore LocalScore details

What fits on the NVIDIA RTX 4080 SUPER

Estimates for this chip are scaled from a measured run: Llama 2 7B Q4_0 at 150 tok/s (Vulkan, source). Fit assumes the whole model in GPU memory with ~8K context.

ModelBest quant that fitsFileExpected tok/s
Qwen3.8-27BUD-IQ3_XXS10.9 GB52.5 estimate
gemma-4-12B-itQ8_012.7 GB45.3 estimate
Qwen3.5-9BQ8_09.5 GB60.3 estimate
Qwen3.5-4BQ8_04.5 GB128 estimate
Qwen3.6-35B-A3BUD-IQ3_XXS13.2 GB254 estimate
gemma-4-26B-A4B-itUD-IQ4_XS13.6 GB137 estimate
MiMo-V2.6-Distill-Qwen-9BQ8_09.5 GB60.1 estimate
gemma-4-E4B-itQ8_08.2 GB70.1 estimate
Muse-Glimmer-30BUD-IQ3_XXS13.1 GB43.7 estimate
Qwen3-4BQ8_04.3 GB134 estimate
gemma-4-31B-itUD-IQ3_XXS11.8 GB48.5 estimate
gemma-4-12B-it-qat-q4_0-unquantizedUD-Q4_K_XL6.7 GB85.5 estimate
gemma-3-1b-itQ8_01.1 GB478 estimate
gemma-4-E2B-itQ8_05.0 GB116 estimate

Open in the picker

Similar hardware

NVIDIA RTX 5080 · NVIDIA RTX 5070 Ti · NVIDIA RTX 4080 · NVIDIA RTX 4070 Ti SUPER · NVIDIA RTX 5060 Ti · NVIDIA RTX PRO 2000 Blackwell · NVIDIA Quadro P5000 · NVIDIA Quadro RTX 5000