LLM Neuroanatomy: Rewriting the Architecture of Thought Without Retraining

The work of David Noel Ng represents a paradigm shift in our understanding of large language model architecture, demonstrating that intelligence in these systems can be enhanced not through parameter expansion or fine-tuning, but through surgical manipulation of their internal cognitive structure. Ng's achievement of topping the HuggingFace Open LLM Leaderboard without modifying a single weight offers profound insights into how transformers organize themselves during training and suggests entirely new approaches to scaling AI capabilities.

At the heart of Ng's discovery lies the counterintuitive observation that duplicating specific blocks of middle layers in a pre-trained model can significantly enhance performance across diverse cognitive tasks. By creating a configuration where seven middle layers (45-51) in a Qwen2-72B model were traversed twice during inference, Ng produced RYS-XLarge, a model that improved on five out of six benchmarks while maintaining competitive performance on the sixth. This achievement is particularly remarkable given that the optimization was performed using only two RTX 4090 GPUs and guided by narrow proxy tasks—hard mathematical problems and emotional intelligence assessments—that bore no direct resemblance to the leaderboard metrics.

The significance of this approach extends beyond mere performance improvements. Ng's "brain scanner" methodology, which systematically tested thousands of layer configurations, reveals that transformer models develop a genuine functional anatomy. Early layers appear to encode input into abstract representations, while late layers decode back to output format. The middle layers, what Ng terms the "reasoning cortex," operate in a universal internal language robust to architectural rearrangement. This functional anatomy explains why certain layer duplications work while others fail: the reasoning cortex is organized into discrete, multi-layer circuits rather than interchangeable processing units.

The discovery that only circuit-sized blocks of layers can be effectively duplicated challenges conventional wisdom about model scaling. Single layer duplications consistently degraded performance, suggesting that transformer reasoning operates as coherent multi-step processes rather than independent operations. This finding has profound implications for both mechanistic interpretability and model design. If transformers develop specialized circuits during training, then enhancing their capabilities might require understanding and augmenting these circuits rather than simply adding parameters.

Ng's work also suggests a critical mass hypothesis for model development. Smaller models appear to have more entangled functional anatomy, where encoding, reasoning, and decoding functions are distributed throughout the layer stack. Larger models like the 72B parameter Qwen2 used in these experiments develop more separated anatomical structures, with distinct regions for different cognitive functions. This separation might explain why layer duplication proved particularly effective with larger models—there exists sufficient "space" for generalized reasoning circuits to develop.

The technique's orthogonality to fine-tuning presents another intriguing dimension. While layer duplication modifies the architecture without changing weights, fine-tuning modifies weights without changing architecture. The subsequent success of models that combine both approaches—such as MaziyarPanahi's calme-2.4-rys-78b and dfurman's CalmeRys-78B-Orpo-v0.1—suggests these techniques address complementary aspects of model capability. This opens the door to novel hybrid approaches that might extract maximum performance from existing architectures.

The implications of this research extend beyond technical performance to fundamental questions about artificial cognition. Ng's observation that improperly duplicated layers can produce models with "bizarre personality disorders" or degenerate conversational patterns suggests that transformers possess something akin to functional specialization. If these models can be said to have a "state of mind," then layer manipulation might represent a form of artificial neurosurgery—selectively enhancing or disrupting specific cognitive functions.

From a practical perspective, the layer duplication technique offers a computationally efficient path to model enhancement. By avoiding the need for retraining, it dramatically reduces the computational resources required to improve model performance. The method's effectiveness across diverse benchmarks also suggests a form of emergent generalization—improvements in narrow proxy tasks transfer broadly to unseen capabilities.

Despite these promising findings, several questions remain unanswered. The technique appears model-specific, with different architectures requiring different layer configurations. The closure of the HuggingFace leaderboard has also made systematic validation more challenging. Additionally, the long-term implications of repeatedly traversing specific layers—particularly regarding potential amplification of biases or limitations—warrants further investigation.

Ng's work ultimately represents a new frontier in AI research: one that moves beyond treating large language models as black boxes to understanding and manipulating their internal cognitive architecture. As AI systems continue to grow in capability and complexity, such approaches may prove essential not just for performance enhancement, but for understanding the fundamental nature of artificial cognition itself. The ability to "perform brain surgery on artificial minds," as Ng puts it, may one day allow us to create AI systems with more refined, specialized, and perhaps even more human-like cognitive abilities.

For those interested in exploring this technique further, Ng has indicated plans to release code and additional RYS models for newer architectures like Qwen3.5 27B. The method's accessibility—requiring only inference capabilities rather than training infrastructure—makes it particularly valuable for researchers and developers with limited computational resources. As the field continues to evolve, Ng's discovery of LLM neuroanatomy may prove to be not just a clever optimization technique, but a key insight into the fundamental organization of artificial thought.