NanoGPT Slowrun: Rethinking AI Scaling Laws Through Data Efficiency

The AI scaling paradigm is hitting a wall. While compute doubles every 18 months, the data required to train ever-larger models isn't keeping pace. This asymmetry threatens to bottleneck progress in artificial intelligence, regardless of how many GPUs we throw at the problem.

Enter NanoGPT Slowrun, an open competition launched by Q Labs that flips the traditional optimization script. Instead of racing for speed, Slowrun optimizes for data efficiency—training on a fixed 100 million tokens from the FineWeb dataset while allowing unlimited compute. The goal: achieve the lowest validation loss possible.

The Compute-Data Bottleneck

The fundamental issue is straightforward but profound. Current scaling laws demand proportional increases in both compute and data. Yet compute grows exponentially while high-quality data remains stubbornly finite. This creates an inevitable bottleneck.

As Q Labs observes, this limitation is visible across domains. In robotics and biology, practitioners would eagerly deploy 1000x more compute if it yielded significantly better results. But without corresponding data growth, additional compute yields diminishing returns.

"Intelligence will eventually be bottlenecked by data, not compute," notes the Q Labs team. This insight drives their mission to develop learning algorithms that excel in "limited data, practically infinite compute" settings.

How Slowrun Works

The competition mechanics are elegantly simple. Participants train models on 100M FineWeb tokens, using as much compute as desired. Improvements are submitted via pull requests to the NanoGPT Slowrun repository and merged if they lower validation loss.

This constraint—optimizing for data efficiency rather than speed—creates space for ideas typically filtered out by traditional benchmarks. Heavy regularization, second-order optimizers, and gradient descent alternatives become viable strategies rather than computational luxuries.

Early Results: 5.5x Data Efficiency

In just one week, community contributions have pushed data efficiency from ~2.4x to 5.5x against the modded-nanogpt baseline—more than doubling performance in days.

The key breakthroughs include:

• Epoch shuffling: Shuffling data at the start of each epoch had outsized impact on multi-epoch training
• Learned projections: Replacing separate embedding tables with learned projections for value embeddings
• Activation functions: Swapping squared ReLU for SwiGLU activation
• Model ensembling: Combining multiple models for improved performance

Muon optimizer emerged as a standout performer, outperforming established optimizers like AdamW, SOAP, and MAGMA. Multi-epoch training proved essential, and aggressive regularization—weight decay up to 16x standard plus dropout—enabled successful scaling to larger parameter counts.

The Road Ahead

The team believes 10x data efficiency is reachable in the short term. The 100x milestone, while ambitious, might be feasible by year's end given the unexplored algorithmic directions.

Several promising avenues remain wide open:

• Second-order optimizers and natural gradient methods
• Diffusion models for training efficiency
• Curriculum learning strategies
• Gradient descent alternatives like evolutionary search
• Compression and model-complexity optimization

Each represents a potential breakthrough in breaking the compute-data symmetry that constrains current AI progress.

Why This Matters

NanoGPT Slowrun isn't just another AI competition—it's an experiment in rethinking how we approach machine learning at scale. By decoupling compute from data constraints, it creates a laboratory for exploring algorithms that might otherwise be computationally prohibitive.

The implications extend beyond academic curiosity. If successful, these approaches could unlock AI capabilities in data-scarce domains, from specialized scientific applications to resource-constrained environments. They might also reveal fundamental principles about generalization that apply across the AI landscape.

As the competition continues, one thing is clear: the future of AI scaling may depend not on finding more data, but on learning to do more with less. NanoGPT Slowrun is betting that the next breakthrough in artificial intelligence won't come from bigger datasets, but from smarter algorithms.

Learn more about NanoGPT Slowrun and contribute to the open effort.