GPUs & Macs › NVIDIA RTX 4000 Ada

NVIDIA RTX 4000 Ada: local LLM speed

Llama 2 7B (Q4_0) generates 61.8 tokens/s on the NVIDIA RTX 4000 Ada (CUDA), and processes prompts at 2,780 tokens/s.

Measured results

ModelGenerate
tok/s		Prompt
tok/s		QuantBackendTestSource
Llama 2 7B61.82,780Q4_0CUDA FA offtg128llama.cpp #15013 details
Llama 2 7B61.43,172Q4_0CUDA FA ontg128llama.cpp #15013 details
Llama 3.2 1B Instruct21010,448Q4_K_Mllamafile LocalScore avgLocalScore details
Llama 3.1 8B Instruct53.52,608Q4_K_Mllamafile LocalScore avgLocalScore details
Qwen2.5 14B Instruct30.01,417Q4_K_Mllamafile LocalScore avgLocalScore details

What fits on the NVIDIA RTX 4000 Ada

Estimates for this chip are scaled from a measured run: Llama 2 7B Q4_0 at 61.8 tok/s (CUDA, source). Fit assumes the whole model in GPU memory with ~8K context.

ModelBest quant that fitsFileExpected tok/s
Qwen3.8-27BUD-Q4_K_M16.5 GB14.4 estimate
gemma-4-12B-itQ8_012.7 GB18.7 estimate
Qwen3.5-9BQ8_09.5 GB24.8 estimate
Qwen3.5-4BQ8_04.5 GB52.7 estimate
Qwen3.6-35B-A3BUD-IQ4_XS17.7 GB77.8 estimate
gemma-4-26B-A4B-itUD-Q4_K_M16.9 GB45.3 estimate
MiMo-V2.6-Distill-Qwen-9BQ8_09.5 GB24.8 estimate
gemma-4-E4B-itQ8_08.2 GB28.8 estimate
Muse-Glimmer-30BUD-IQ3_M14.1 GB16.7 estimate
Qwen3-4BQ8_04.3 GB55.2 estimate
gemma-4-31B-itQ4_K_S17.4 GB13.6 estimate
gemma-4-26B-A4B-it-qat-q4_0-unquantizedUD-Q4_K_XL14.2 GB53.9 estimate
gemma-4-12B-it-qat-q4_0-unquantizedUD-Q4_K_XL6.7 GB35.2 estimate
gemma-3-1b-itQ8_01.1 GB197 estimate

Open in the picker

Similar hardware

NVIDIA RTX A4500 · NVIDIA RTX 4000 SFF Ada · NVIDIA RTX PRO 4000 Blackwell · NVIDIA RTX A5000 · NVIDIA Quadro RTX 6000 · NVIDIA RTX A4000 · NVIDIA RTX 2000 Ada · NVIDIA RTX A2000 12GB