GPUs & Macs

Speed shown: Llama 2 7B Q4_0 generation, best backend.

NVIDIA

• NVIDIA RTX 5090 300 tok/s
• NVIDIA RTX PRO 6000 Blackwell 281 tok/s
• NVIDIA H100 SXM 281 tok/s
• NVIDIA A100 80GB PCIe 201 tok/s
• NVIDIA RTX 5080 192 tok/s
• NVIDIA RTX 4090 D 192 tok/s
• NVIDIA RTX 4090 190 tok/s
• NVIDIA RTX 5070 Ti 182 tok/s
• NVIDIA RTX 6000 Ada 179 tok/s
• NVIDIA RTX 3090 Ti 172 tok/s
• NVIDIA RTX 3090 172 tok/s
• Nvidia A100 171 tok/s
• NVIDIA RTX PRO 4500 Blackwell 169 tok/s
• NVIDIA RTX 5090 Laptop 158 tok/s
• NVIDIA L40 154 tok/s
• NVIDIA RTX 4080 SUPER 150 tok/s
• NVIDIA RTX 3080 146 tok/s
• NVIDIA RTX A6000 145 tok/s
• NVIDIA RTX 4080 143 tok/s
• NVIDIA Tesla V100 32GB 142 tok/s
• NVIDIA RTX A5000 140 tok/s
• NVIDIA RTX PRO 4000 Blackwell 136 tok/s
• NVIDIA RTX 4070 Ti SUPER 136 tok/s
• NVIDIA A30 132 tok/s
• NVIDIA RTX 5070 128 tok/s
• NVIDIA A40 126 tok/s
• NVIDIA RTX 4070 Ti 114 tok/s
• NVIDIA TITAN V 112 tok/s
• NVIDIA RTX 4070 SUPER 112 tok/s
• NVIDIA RTX 2080 Ti 109 tok/s
• NVIDIA Quadro RTX 6000 104 tok/s
• NVIDIA Quadro RTX 8000 104 tok/s
• NVIDIA RTX A4500 103 tok/s
• NVIDIA RTX 5060 Ti 97.0 tok/s
• NVIDIA RTX 4070 92.3 tok/s
• NVIDIA RTX 3060 Ti 90.3 tok/s
• NVIDIA RTX 2070 SUPER 88.1 tok/s
• NVIDIA RTX A4000 85.2 tok/s
• NVIDIA RTX 3060 80.6 tok/s
• NVIDIA RTX 3070 78.7 tok/s
• NVIDIA TITAN Xp 76.1 tok/s
• Nvidia Tesla T10 76.0 tok/s
• NVIDIA GTX 1080 Ti 71.6 tok/s
• NVIDIA Quadro RTX 4000 67.6 tok/s
• NVIDIA Tesla P100 66.5 tok/s
• NVIDIA P102-100 66.5 tok/s
• NVIDIA RTX 4060 Ti 64.0 tok/s
• NVIDIA RTX 3070 Laptop 63.6 tok/s
• NVIDIA RTX 4000 Ada 61.8 tok/s
• NVIDIA RTX 2060 SUPER 61.1 tok/s
• NVIDIA DGX Spark (GB10) 60.4 tok/s
• NVIDIA RTX 4060 Laptop 60.0 tok/s
• NVIDIA Tesla P40 59.4 tok/s
• Nvidia GTX 1660 Ti Laptop 57.3 tok/s
• NVIDIA RTX 2000 Ada 50.7 tok/s
• NVIDIA RTX 3060 Laptop 49.0 tok/s
• NVIDIA P104-100 48.6 tok/s
• NVIDIA Tesla T4 46.4 tok/s
• NVIDIA RTX A2000 45.7 tok/s
• NVIDIA RTX 4050 Laptop 43.7 tok/s
• NVIDIA GTX 1070 Ti 43.4 tok/s
• NVIDIA GTX 1070 43.3 tok/s
• NVIDIA Tesla M40 41.7 tok/s
• NVIDIA GTX 1660 41.4 tok/s
• NVIDIA Jetson AGX Orin 64GB 35.9 tok/s
• NVIDIA GTX 980 34.2 tok/s
• NVIDIA Tesla P4 33.3 tok/s
• NVIDIA P106-100 30.8 tok/s
• NVIDIA GTX 1060 28.2 tok/s
• NVIDIA T1000 27.8 tok/s
• NVIDIA Quadro P2000 23.8 tok/s
• NVIDIA GTX 1050 Ti 21.0 tok/s
• NVIDIA Tesla K80 14.3 tok/s
• NVIDIA Quadro P1000 14.0 tok/s
• Nvidia Tesla K40 9.9 tok/s
• NVIDIA H100 PCIe 3 results
• NVIDIA A100 80GB SXM 3 results
• NVIDIA RTX PRO 6000 Blackwell Max-Q 5 results
• NVIDIA L40S 3 results
• NVIDIA RTX 3080 Ti 3 results
• NVIDIA H100 NVL 3 results
• NVIDIA RTX 5070 Ti Laptop 3 results
• NVIDIA RTX 3070 Ti 2 results
• NVIDIA Tesla V100 16GB 4 results
• NVIDIA TITAN RTX 3 results
• NVIDIA RTX 5070 Laptop 1 results
• NVIDIA RTX 3500 Ada Laptop 3 results
• NVIDIA RTX 3070 Ti Laptop 1 results
• NVIDIA RTX 3080 Laptop 3 results
• NVIDIA RTX 4070 Laptop 3 results
• NVIDIA RTX 2080 SUPER 3 results
• NVIDIA L4 3 results
• NVIDIA RTX 4000 SFF Ada 3 results
• NVIDIA A10 3 results
• NVIDIA RTX 2080 3 results
• NVIDIA RTX A5000 Laptop 3 results
• NVIDIA RTX 4060 6 results
• NVIDIA RTX 2070 2 results
• NVIDIA RTX PRO 2000 Blackwell Laptop 3 results
• NVIDIA RTX 2060 3 results
• NVIDIA RTX 2000 Ada Laptop 1 results
• NVIDIA RTX A2000 12GB 3 results
• NVIDIA RTX 3050 2 results
• NVIDIA RTX 1000 Ada Laptop 1 results
• NVIDIA RTX A2000 8GB Laptop 2 results
• NVIDIA RTX 3050 6GB Laptop 2 results
• NVIDIA RTX PRO 1000 Blackwell Laptop 1 results
• NVIDIA Quadro RTX 3000 1 results
• NVIDIA GTX 1080 2 results
• NVIDIA RTX 3050 Laptop 2 results
• NVIDIA RTX 3050 Ti Laptop 2 results
• NVIDIA GTX 1660 SUPER 2 results
• NVIDIA T1000 8GB 1 results
• NVIDIA GTX 1650 Ti 2 results
• NVIDIA Quadro P620 2 results
• NVIDIA A40-48Q 1 results
• NVIDIA RTX PRO 6000 Blackwell Server Edition 2 results
• NVIDIA RTX PRO 2000 Blackwell 1 results
• NVIDIA RTX 5000 Ada Laptop 1 results
• NVIDIA Quadro P5000 1 results
• NVIDIA RTX A4500 Laptop 2 results
• NVIDIA L40S-8Q 2 results
• NVIDIA A2 1 results
• NVIDIA RTX 4090 Laptop 1 results
• NVIDIA H200 1 results
• NVIDIA Quadro RTX 5000 1 results
• NVIDIA RTX A1000 Laptop 1 results
• NVIDIA RTX 5060 1 results
• NVIDIA RTX 5080 Laptop 1 results
• NVIDIA RTX A2000 Laptop 1 results
• NVIDIA RTX 500 Ada Laptop 1 results
• NVIDIA Quadro P6000 1 results
• NVIDIA Quadro P4000 1 results
• NVIDIA RTX 2050 1 results
• NVIDIA P106-090 1 results
• NVIDIA GTX 1660 Ti 1 results
• NVIDIA GTX 1050 1 results
• NVIDIA GTX 1070 Max-Q 1 results
• NVIDIA GTX 1650 SUPER 1 results
• NVIDIA GT 1030 1 results
• NVIDIA Quadro T2000 1 results
• NVIDIA T1200 Laptop 1 results
• NVIDIA GTX 1650 1 results
• NVIDIA GTX 1650 Ti Max-Q 1 results
• NVIDIA MX150 1 results

AMD

• AMD Instinct MI300X 233 tok/s
• AMD Radeon RX 7900 XTX 191 tok/s
• AMD Radeon AI PRO R9700 153 tok/s
• AMD Radeon RX 9070 XT 137 tok/s
• AMD Instinct MI210 130 tok/s
• AMD Radeon Pro W7900 127 tok/s
• AMD Radeon RX 7800 XT 125 tok/s
• AMD Radeon RX 7900 XT 123 tok/s
• AMD Radeon RX 9070 120 tok/s
• AMD Instinct MI50 118 tok/s
• AMD Radeon RX 7900 GRE 116 tok/s
• AMD Radeon VII 111 tok/s
• AMD Instinct MI100 110 tok/s
• AMD Radeon Pro VII 108 tok/s
• AMD Radeon RX 6900 XT 108 tok/s
• AMD Radeon RX 6800 106 tok/s
• AMD Radeon RX 6800 XT 100 tok/s
• AMD Radeon Pro W6800X Duo 100 tok/s
• AMD Instinct MI60 91.5 tok/s
• AMD Radeon Pro W6800X 86.5 tok/s
• AMD Radeon RX 6700 XT 83.9 tok/s
• AMD Radeon RX 6750 XT 82.3 tok/s
• AMD Radeon Pro V620 81.8 tok/s
• AMD Radeon RX 5700 XT 74.4 tok/s
• AMD Radeon RX 9060 XT 74.0 tok/s
• AMD Radeon Pro W5700 71.3 tok/s
• AMD Radeon Pro V540 69.6 tok/s
• AMD Radeon RX Vega 64 67.7 tok/s
• AMD Radeon RX 6650 XT 64.5 tok/s
• AMD Instinct MI25 63.9 tok/s
• AMD BC-250 63.1 tok/s
• AMD Radeon RX 7600 60.9 tok/s
• AMD Radeon RX 7600 XT 56.1 tok/s
• AMD FirePro S9300 x2 55.6 tok/s
• AMD Ryzen AI Max+ 395 (Radeon 8060S) 54.4 tok/s
• AMD Radeon RX 6600 XT 53.9 tok/s
• AMD Radeon RX 6600 53.2 tok/s
• AMD Radeon RX Vega 56 52.7 tok/s
• AMD Radeon RX 6800M 49.1 tok/s
• AMD Radeon RX 6600M 48.2 tok/s
• AMD Radeon Pro V340 47.5 tok/s
• AMD Radeon RX 7600M XT 45.3 tok/s
• AMD Radeon RX 470 43.4 tok/s
• AMD Radeon RX 570 40.2 tok/s
• AMD Radeon RX 580 39.3 tok/s
• AMD Radeon RX 480 37.2 tok/s
• AMD Radeon Pro W5500 36.8 tok/s
• AMD FirePro W8100 29.5 tok/s
• AMD Radeon RX 6500 XT 27.8 tok/s
• AMD FirePro S10000 25.3 tok/s
• AMD Ryzen AI 300 Series 23.9 tok/s
• AMD Ryzen 8000 Series 21.3 tok/s
• AMD Ryzen 6000 Series 21.3 tok/s
• AMD Ryzen 7000 Series 20.1 tok/s
• AMD Ryzen AI 400 Series 19.0 tok/s
• AMD Ryzen Z1 Extreme 18.8 tok/s
• AMD Steam Deck 18.4 tok/s
• AMD FirePro D700 16.6 tok/s
• AMD Radeon Pro WX 4100 16.6 tok/s
• AMD Radeon 880M 16.2 tok/s
• AMD Ryzen 5000 Series 11.0 tok/s
• AMD Radeon RX 550 10.2 tok/s
• AMD Ryzen 4000 Series 9.9 tok/s
• AMD Ryzen 3000 Series 8.5 tok/s
• AMD Ryzen 5 3000 Series 8.5 tok/s
• AMD Radeon RX 6950 XT 2 results
• AMD Radeon RX 7700S 1 results
• AMD Radeon 780M 4 results
• AMD Radeon 760M 1 results
• AMD Radeon RX 5500 XT 1 results
• AMD Radeon 740M 1 results

Apple

• Apple M5 Ultra (80-core GPU) 179 tok/s
• Apple M5 Max (40-core GPU) 120 tok/s
• Apple M3 Ultra 116 tok/s
• Apple M2 Ultra (76-core GPU) 94.3 tok/s
• Apple M3 Ultra (80-core GPU) 92.1 tok/s
• Apple M2 Ultra (60-core GPU) 88.6 tok/s
• Apple M3 Ultra (60-core GPU) 88.4 tok/s
• Apple M1 Ultra (64-core GPU) 83.7 tok/s
• Apple M4 Max (40-core GPU) 83.1 tok/s
• Apple M4 Max 75.0 tok/s
• Apple M1 Ultra (48-core GPU) 74.9 tok/s
• Apple M4 Max (32-core GPU) 70.0 tok/s
• Apple M5 Pro (16-core GPU) 67.6 tok/s
• Apple M5 Pro (20-core GPU) 66.3 tok/s
• Apple M3 Max (40-core GPU) 66.3 tok/s
• Apple M2 Max (38-core GPU) 66.0 tok/s
• Apple M1 Max (32-core GPU) 61.2 tok/s
• Apple M2 Max (30-core GPU) 61.0 tok/s
• Apple M3 Max (30-core GPU) 56.6 tok/s
• Apple M1 Max (24-core GPU) 54.6 tok/s
• Apple M4 Pro (20-core GPU) 50.7 tok/s
• Apple M4 Pro (16-core GPU) 49.6 tok/s
• Apple M2 Pro (19-core GPU) 38.9 tok/s
• Apple M2 Pro (16-core GPU) 37.9 tok/s
• Apple M1 Pro (16-core GPU) 36.4 tok/s
• Apple M2 Ultra 36.1 tok/s
• Apple M6 (8-core GPU) 35.7 tok/s
• Apple M1 Pro (14-core GPU) 35.5 tok/s
• Apple M5 (10-core GPU) 31.9 tok/s
• Apple M3 Pro (18-core GPU) 30.7 tok/s
• Apple M3 Pro (14-core GPU) 30.6 tok/s
• Apple M3 26.4 tok/s
• Apple M4 (10-core GPU) 24.1 tok/s
• Apple M2 (10-core GPU) 21.9 tok/s
• Apple M2 (8-core GPU) 21.7 tok/s
• Apple M3 (10-core GPU) 21.3 tok/s
• Apple M2 Pro 21.0 tok/s
• Apple M1 (7-core GPU) 14.2 tok/s
• Apple M1 (8-core GPU) 14.2 tok/s
• Apple M2 13.5 tok/s
• Apple M1 12.8 tok/s
• Apple M5 Max (32-core GPU) 3 results
• Apple M6 (12-core GPU) 1 results
• Apple M4 (8-core GPU) 3 results
• Apple M3 (8-core GPU) 2 results
• Apple A18 Pro (5-core GPU) 2 results
• Apple M1 Max 2 results
• Apple M1 Pro 1 results

Intel

• Intel Arc Pro B70 114 tok/s
• Intel Arc B580 72.0 tok/s
• Intel Arc Pro B60 70.5 tok/s
• Intel Arc A770 53.1 tok/s
• Intel Arc B570 50.5 tok/s
• Intel Arc A770M 47.7 tok/s
• Intel Arc Pro B50 47.7 tok/s
• Intel Arc A750 44.0 tok/s
• Intel Core Ultra 200 Series 25.1 tok/s
• Intel Core Ultra 300 Series 24.4 tok/s
• Intel Core 1100 Series 10.6 tok/s
• Intel Core Ultra 100 Series 8.2 tok/s
• Intel Core 1000 Series 4.2 tok/s
• Intel Core 8000 Series 3.4 tok/s
• Intel N150 2.9 tok/s

Other

• MediaTek Dimensity 9400 8.9 tok/s
• CIX CD8180 5.5 tok/s