GPUs & Macs › NVIDIA Quadro RTX 8000

NVIDIA Quadro RTX 8000: local LLM speed

Llama 2 7B (Q4_0) generates 104 tokens/s on the NVIDIA Quadro RTX 8000 (CUDA), and processes prompts at 3,052 tokens/s.

Measured results

ModelGenerate
tok/s		Prompt
tok/s		QuantBackendTestSource
Llama 2 7B1043,052Q4_0CUDA FA ontg128llama.cpp #15013 details
Llama 2 7B1032,710Q4_0CUDA FA offtg128llama.cpp #15013 details
Llama 3.2 1B Instruct2348,780Q4_K_Mllamafile LocalScore avg · n=2LocalScore LocalScore details
Llama 3.1 8B Instruct74.01,677Q4_K_Mllamafile LocalScore avgLocalScore details
Qwen2.5 14B Instruct39.81,132Q4_K_Mllamafile LocalScore avg · n=2LocalScore LocalScore details

What fits on the NVIDIA Quadro RTX 8000

Estimates for this chip are scaled from a measured run: Llama 2 7B Q4_0 at 104 tok/s (CUDA, source). Fit assumes the whole model in GPU memory with ~8K context.

ModelBest quant that fitsFileExpected tok/s
Qwen3.8-27BQ8_029.0 GB13.6 estimate
gemma-4-12B-itQ8_012.7 GB31.3 estimate
Qwen3.5-9BQ8_09.5 GB41.6 estimate
Qwen3.5-4BQ8_04.5 GB88.4 estimate
Qwen3.6-35B-A3BQ8_036.9 GB62.6 estimate
gemma-4-26B-A4B-itQ8_026.9 GB47.9 estimate
MiMo-V2.6-Distill-Qwen-9BQ8_09.5 GB41.5 estimate
gemma-4-E4B-itQ8_08.2 GB48.3 estimate
Muse-Glimmer-30BQ8_029.6 GB13.4 estimate
Qwen3-4BQ8_04.3 GB92.5 estimate
gemma-4-31B-itQ8_032.6 GB12.1 estimate
gemma-4-26B-A4B-it-qat-q4_0-unquantizedUD-Q4_K_XL14.2 GB90.4 estimate
gemma-4-12B-it-qat-q4_0-unquantizedUD-Q4_K_XL6.7 GB59.0 estimate
gemma-3-1b-itQ8_01.1 GB330 estimate

Open in the picker

Similar hardware

NVIDIA RTX 6000 Ada · NVIDIA RTX A6000 · NVIDIA RTX PRO 4500 Blackwell · NVIDIA RTX PRO 4000 Blackwell · NVIDIA RTX A5000 · NVIDIA Quadro RTX 6000 · NVIDIA RTX A4500 · NVIDIA RTX 4000 Ada