GPUs & Macs › NVIDIA RTX PRO 6000 Blackwell Max-Q

NVIDIA RTX PRO 6000 Blackwell Max-Q: local LLM speed

5 measured results for the NVIDIA RTX PRO 6000 Blackwell Max-Q.

Measured results

ModelGenerate
tok/s		Prompt
tok/s		QuantBackendTestSource
gpt-oss 20B2509,481MXFP4CUDA FA ontg128llama.cpp #15396 details
gpt-oss 120B1845,449MXFP4CUDA FA ontg128 · n=2llama.cpp #15396 llama.cpp #15396 details
Llama 3.2 1B Instruct10616,184Q4_K_Mllamafile LocalScore avgLocalScore details
Llama 3.1 8B Instruct46.25,544Q4_K_Mllamafile LocalScore avgLocalScore details
Qwen2.5 14B Instruct23.72,623Q4_K_Mllamafile LocalScore avgLocalScore details

What fits on the NVIDIA RTX PRO 6000 Blackwell Max-Q

Estimates for this chip are scaled from a measured run: Llama 3.1 8B Instruct Q4_K_M at 46.2 tok/s (llamafile, source). Fit assumes the whole model in GPU memory with ~8K context.

ModelBest quant that fitsFileExpected tok/s
Qwen3.8-27BQ8_029.0 GB7.8 estimate
gemma-4-12B-itQ8_012.7 GB17.9 estimate
Qwen3.5-9BQ8_09.5 GB23.9 estimate
Qwen3.5-4BQ8_04.5 GB50.7 estimate
Qwen3.6-35B-A3BQ8_036.9 GB36.0 estimate
Qwen3.8-Flash-NextUD-IQ3_XXS82.0 GB2.8 estimate
gemma-4-26B-A4B-itQ8_026.9 GB27.5 estimate
MiMo-V2.6-Distill-Qwen-9BQ8_09.5 GB23.8 estimate
gemma-4-E4B-itQ8_08.2 GB27.8 estimate
Muse-Glimmer-30BQ8_029.6 GB7.7 estimate
Qwen3-4BQ8_04.3 GB53.1 estimate
gemma-4-31B-itQ8_032.6 GB7.0 estimate
gemma-4-26B-A4B-it-qat-q4_0-unquantizedUD-Q4_K_XL14.2 GB51.9 estimate
gemma-4-12B-it-qat-q4_0-unquantizedUD-Q4_K_XL6.7 GB33.9 estimate

Open in the picker

Similar hardware

NVIDIA RTX PRO 6000 Blackwell · NVIDIA RTX 6000 Ada · NVIDIA RTX A6000 · NVIDIA Quadro RTX 8000 · NVIDIA RTX PRO 4500 Blackwell · NVIDIA RTX PRO 4000 Blackwell · NVIDIA RTX A5000 · NVIDIA Quadro RTX 6000