GPUs & Macs › NVIDIA RTX 4090

NVIDIA RTX 4090: local LLM speed

Llama 2 7B (Q4_0) generates 190 tokens/s on the NVIDIA RTX 4090 (Vulkan), and processes prompts at 10,830 tokens/s.

Measured results

ModelGenerate
tok/s		Prompt
tok/s		QuantBackendTestSource
Llama 2 7B19010,830Q4_0Vulkan FA ontg128llama.cpp #10879 details
Llama 2 7B18914,771Q4_0CUDA FA ontg128llama.cpp #15013 details
Llama 2 7B1889,452Q4_0Vulkan FA offtg128llama.cpp #10879 details
Llama 2 7B18611,993Q4_0CUDA FA offtg128llama.cpp #15013 details
Llama 3.2 1B Instruct29222,429Q4_K_Mllamafile LocalScore avg · n=2LocalScore LocalScore details
gpt-oss 20B2559,194MXFP4CUDA FA ontg128 · n=3llama.cpp #15396 llama.cpp #15396 details
Llama 3.1 8B Instruct1057,146Q4_K_Mllamafile LocalScore avg · n=2LocalScore LocalScore details
Qwen2.5 14B Instruct49.13,506Q4_K_Mllamafile LocalScore avg · n=3LocalScore LocalScore LocalScore details

What fits on the NVIDIA RTX 4090

Estimates for this chip are scaled from a measured run: Llama 2 7B Q4_0 at 190 tok/s (Vulkan, source). Fit assumes the whole model in GPU memory with ~8K context.

ModelBest quant that fitsFileExpected tok/s
Qwen3.8-27BUD-Q6_K22.0 GB33.1 estimate
gemma-4-12B-itQ8_012.7 GB57.4 estimate
Qwen3.5-9BQ8_09.5 GB76.3 estimate
Qwen3.5-4BQ8_04.5 GB162 estimate
Qwen3.6-35B-A3BUD-Q4_K_S20.9 GB203 estimate
gemma-4-26B-A4B-itUD-Q5_K_M21.2 GB112 estimate
MiMo-V2.6-Distill-Qwen-9BQ8_09.5 GB76.1 estimate
gemma-4-E4B-itQ8_08.2 GB88.7 estimate
Muse-Glimmer-30BUD-Q5_K_M19.2 GB37.9 estimate
Qwen3-4BQ8_04.3 GB170 estimate
gemma-4-31B-itQ5_K_M21.7 GB33.6 estimate
gemma-4-26B-A4B-it-qat-q4_0-unquantizedUD-Q4_K_XL14.2 GB166 estimate
gemma-4-12B-it-qat-q4_0-unquantizedUD-Q4_K_XL6.7 GB108 estimate
gemma-3-1b-itQ8_01.1 GB606 estimate

Open in the picker

Similar hardware

NVIDIA RTX 4090 D · NVIDIA RTX 3090 Ti · NVIDIA RTX 3090 · NVIDIA TITAN RTX · NVIDIA Quadro P6000 · NVIDIA RTX 5090 · NVIDIA RTX 5080 · NVIDIA RTX 5070 Ti