Andrej Karpathy Achieves 4.3% Speedup in GPT-2 Training with fp8 Precision

Andrej Karpathy, the renowned AI researcher and former Tesla AI director, has achieved a notable breakthrough in large language model training efficiency. By enabling fp8 (8-bit floating point) training, Karpathy reports a 4.3% improvement in "time to GPT-2," reducing the training time to just 2.91 hours. Even more impressively, when using 8XH100 spot instance prices, the entire GPT-2 reproduction now costs only approximately $20.

This development carries significant implications for the AI industry. As Karpathy wryly notes, "GPT-2 (7 years ago): too dangerous to release. GPT-2 (today): new MNIST! :)" The reference to MNIST—the classic introductory dataset for machine learning—underscores how dramatically the landscape has shifted. What was once considered so potentially dangerous that OpenAI withheld the full model is now accessible for the price of a modest dinner.

The Technical Journey

The path to achieving these efficiency gains wasn't straightforward. Karpathy describes fp8 training as "a little bit more tricky than I anticipated," requiring substantial experimentation to implement effectively. While the H100's fp8 capabilities theoretically offer 2X the FLOPS compared to bf16, real-world performance gains proved more modest.

Several factors limited the expected speedup:

• Added overhead from scale conversions
• GEMM operations not large enough on GPT-2 scale to justify the overhead
• Lower precision resulting in smaller quality per training step
• Network overhead and less overall support for fp8

Karpathy experimented with different scaling approaches, finding that row-wise scaling produced loss curves close to bf16 but with net slower stepping, while tensor-wise scaling showed more separation in loss curves (indicating lower quality steps) but achieved approximately 7.3% speedup.

The Cost Revolution

Perhaps the most striking aspect of this work is the dramatic cost reduction. Karpathy's "nanochat" project can now train a GPT-2 grade LLM for under $100—specifically around $73 for 3 hours on a single 8XH100 node. This represents a fundamental shift in accessibility.

For context, GPT-2 was originally developed by OpenAI in 2019 with 1.5 billion parameters. At the time, training such a model required substantial computational resources and was considered a significant achievement. Today, that same capability can be reproduced on commodity cloud infrastructure for less than the cost of many consumer electronics.

The Future of fp8 Training

Despite the current limitations, Karpathy sees potential for further improvements. He suggests that selective application of fp8 to specific layers, combined with more careful numerical handling throughout the network, could yield better results. The contrast with torchao's reported 25% speedup for Llama3-8B training (versus Karpathy's 7.3% for GPT-2) suggests that model scale plays a crucial role in realizing fp8 benefits.

Karpathy's work demonstrates that the frontier of AI efficiency continues to advance rapidly. As training costs plummet and techniques mature, capabilities once reserved for well-funded research labs become increasingly accessible. The $20 GPT-2 represents not just a technical achievement but a milestone in the democratization of AI capabilities—one that raises important questions about the future trajectory of the field and how society will adapt to increasingly powerful tools that are simultaneously more capable and more affordable.

{{IMAGE:1}}