Models › Llama-3.3-70B-Instruct

Llama-3.3-70B-Instruct: VRAM needs and speed by GPU

At Q4_K_M, Llama-3.3-70B-Instruct needs about 45.1 GB of VRAM. No measured runs yet. Speeds below are estimates. Submit a run.

70.55B parameters · license: llama3.3 · 131,072-token context · GGUF published Dec 6, 2024 · GGUF files on Hugging Face

Memory needed per quantization

QuantFile sizeNeeds (estimate)Smallest card it fits
UD-IQ1_S15.9 GB18.4 GB24 GB cards
UD-IQ1_M17.1 GB19.7 GB24 GB cards
UD-IQ2_XXS19.4 GB22.0 GB24 GB cards
UD-IQ2_M24.3 GB26.9 GB32 GB cards
Q2_K26.4 GB28.9 GB32 GB cards
Q2_K_L26.6 GB29.2 GB32 GB cards
UD-Q2_K_XL27.0 GB29.5 GB32 GB cards
UD-IQ3_XXS27.7 GB30.2 GB32 GB cards
Q3_K_S30.9 GB33.5 GB48 GB cards
Q3_K_M34.3 GB36.8 GB48 GB cards
UD-Q3_K_XL34.8 GB37.3 GB48 GB cards
IQ4_XS37.9 GB40.5 GB48 GB cards
IQ4_NL40.1 GB42.6 GB48 GB cards
Q4_040.1 GB42.7 GB48 GB cards
Q4_K_S40.3 GB42.9 GB48 GB cards
Q4_K_M42.5 GB45.1 GB48 GB cards
UD-Q4_K_XL42.7 GB45.2 GB48 GB cards
Q4_144.3 GB46.9 GB48 GB cards
Q5_K_S48.7 GB51.2 GB96 GB cards
UD-Q5_K_XL49.9 GB52.4 GB96 GB cards
Q5_K_M49.9 GB52.5 GB96 GB cards
UD-Q6_K_XL61.2 GB63.7 GB96 GB cards
Q8_075.0 GB77.5 GB96 GB cards
UD-Q8_K_XL81.2 GB83.8 GB96 GB cards
Q6_K115.8 GB118.3 GBmulti-GPU / large unified memory
BF16141.1 GB143.7 GBmulti-GPU / large unified memory
F16141.1 GB143.7 GBmulti-GPU / large unified memory

Expected speed on popular hardware

HardwareBest quant that fitsExpected tok/s
NVIDIA RTX 3060Does not fit
NVIDIA RTX 4060 TiDoes not fit
NVIDIA RTX 3090UD-IQ2_XXS33.8 estimate
NVIDIA RTX 4090UD-IQ2_XXS37.4 estimate
NVIDIA RTX 5090UD-IQ3_XXS41.5 estimate
AMD Radeon RX 7900 XTXUD-IQ2_XXS37.6 estimate
Apple M4 Pro (20-core GPU) (64 GB)Q4_K_M5.4 estimate
Apple M3 Max (40-core GPU) (64 GB)Q4_K_M8.0 estimate
AMD Ryzen AI Max+ 395 (Radeon 8060S)Q8_02.8 estimate

Check your own card

Too big for your card?

Run Llama-3.3-70B-Instruct on a rented 24-80 GB GPU by the hour.

Rent a bigger GPU on RunPod

Affiliate link: if you sign up, RigBench may earn a referral credit. It never affects rankings or numbers.