Mac GPT GPU Benchmark Explorer

Live Script that benchmarks a small GPT on Apple Silicon with MATLAB-CPU, PyTorch-CPU/MPS, and MLX.

GPT training is usually relegated to GPUs, often intentionally without full floating point precision. MATLAB has no GPU backend on Apple Silicon: gpuArray and the Deep Learning Toolbox training path require an NVIDIA CUDA GPU, which Macs do not have. Every M-series Mac carries an extremely capable GPU of reduced precision but that may be irrelevant in GPT training. MATLAB needs some indirect path to access it.

This educational Live Script benchmarks a small but flexible MATLAB GPT model using the CPU, PyTorch on the CPU, and PyTorch on the Metal GPU (the MPS backend). It also pits Apple's native MLX framework against PyTorch-MPS. It can serve as an introduction to these technologies and to some pitfalls of benchmarking.

The small GPT is arithGPT from nanoGPTArithmetic Explorer and the task is addition where the hard part is learning to carry. This GPT is flexible in size so enables benchmarking at different scales with grokking to 100% exact-match on held-out problems as a performance check.

The tests documented here are for an arguably ancient M1 Chip MacBook Pro. The Live Script package contains instructions for how to setup the environment and perform the same tests on any Mac, e.g. the latest with the M5 chip. New results are saved without clobbering ships results.