OpenMythos: Theoretical Reconstruction of Claude Mythos Architecture Reveals Revolutionary Loop Transformer Design

The open-source community has made significant strides in reverse-engineering advanced AI architectures with the release of OpenMythos, a theoretical reconstruction of what might power Anthropic's Claude Mythos model. This ambitious project posits that Mythos likely operates on a fundamentally different paradigm than conventional transformers—one that leverages recurrence and depth in ways that could dramatically improve reasoning capabilities without proportional increases in computational requirements.

The Central Hypothesis: Recurrent-Depth Transformers

At the heart of OpenMythos lies a compelling hypothesis: Claude Mythos is not simply a larger transformer with more parameters, but rather a Recurrent-Depth Transformer (RDT), also known as a Looped Transformer. Rather than stacking hundreds of unique layers, this architecture reuses a subset of layers multiple times per forward pass, achieving greater effective depth with significantly fewer parameters.

This approach represents a fundamental shift from the current trajectory of AI development, which has primarily focused on scaling model size. As the project's documentation explains, "Same weights. More loops. Deeper thinking. This is not chain-of-thought. There is no intermediate token output. All of this reasoning happens silently, inside a single forward pass, in continuous latent space."

Three-Stage Architecture Design

The OpenMythos implementation divides the model into three functional blocks:

1. Prelude: Standard transformer layers that process input once
2. Recurrent Block: A looped section that runs multiple times, updating hidden states with each iteration
3. Coda: Final standard transformer layers that process the accumulated reasoning

The mathematical elegance of this design is captured in the recurrent block update rule: h_{t+1} = A·h_t + B·e + Transformer(h_t, e), where h_t represents the hidden state after loop t, e is the encoded input injected at every step, and A and B are learned injection parameters. The crucial insight here is that input injection at every loop prevents the model from drifting, maintaining the original signal throughout the reasoning process.

Why This Explains Mythos's Capabilities

The OpenMythos framework provides several compelling explanations for why Claude Mythos demonstrates such exceptional reasoning capabilities:

Systematic Generalization: Conventional transformers struggle with combining knowledge in novel ways not seen during training. Looped transformers, however, appear to undergo a three-stage grokking process: memorization, in-distribution generalization, and finally abrupt systematic generalization to out-of-distribution compositions. This phase transition could explain why Mythos performs qualitatively different from other models on novel questions.

Depth Extrapolation: A remarkable capability of looped transformers is their ability to extrapolate reasoning depth. Training on 5-hop reasoning chains and testing on 10-hop problems becomes possible simply by running more inference-time loops. This directly maps to observations that Mythos handles deeply compositional problems without explicit chain-of-thought.

Latent Thoughts as Implicit Chain-of-Thought: Each loop iteration functions as a step of chain-of-thought, but operates in continuous latent space rather than token space. This allows the model to explore multiple reasoning paths simultaneously, something more akin to breadth-first search than the linear progression of traditional chain-of-thought.

Parameter Efficiency: Perhaps most importantly, a looped model with k layers run L times achieves the quality of a kL-layer non-looped model using only k layers worth of parameters. This means that deeper reasoning becomes "free" in terms of parameters, with memory footprint remaining constant regardless of reasoning depth, and inference-time compute scaling only with loop count.

The Stability Problem and Its Elegant Solution

Training looped models presents significant challenges, primarily instability through residual explosion and sudden loss spikes. The OpenMythos team approaches this through a dynamical systems framework, treating the looping as a discrete linear time-invariant (LTI) system over the residual stream.

The stability of this system is governed by the spectral radius of matrix A (ρ(A)). When ρ(A) < 1, the system remains stable and convergent; when ρ(A) ≥ 1, it becomes unstable and divergent. The solution involves constraining injection parameters to guarantee stability by construction:

"Parameterize A as a continuous negative diagonal matrix, discretize using ZOH/Euler schemes, and enforce negativity via A := Diag(-exp(log_A)) with a learned scalar Δt. This ensures ρ(A) < 1 always holds, regardless of learning rate or batch noise."

This approach, referred to as the "Parcae architecture," represents a likely solution Anthropic employed to make Mythos trainable, enabling robust training even at high learning rates.

Scaling Laws for Looped Models

OpenMythos introduces the first predictable scaling laws specifically for looped training:

• Training: For a fixed FLOP budget with fixed parameters, increasing mean recurrence while reducing token count yields lower loss than training with minimal loops on more data
• Inference: More test-time loops improve quality following a predictable, saturating exponential decay

Empirically, at 770M parameters, a looped model achieves the downstream quality of a 1.3B fixed-depth Transformer trained on the same data—roughly half the parameters for equivalent performance. Applied to Mythos, this suggests that its apparent capability might come substantially from loop depth rather than raw parameter count.

Advanced Implementation Details

The OpenMythos implementation includes several sophisticated features:

Attention Mechanisms: The model supports two attention types:

• GQA (Grouped Query Attention) with optional Flash Attention 2 support for memory efficiency
• MLA (Multi-Latent Attention) with compressed KV latent cache for position-aware compression

Mixture of Experts: The suspected design replaces every FFN in the Recurrent Block with a fine-grained MoE layer, where each FFN is split into many small experts, a router selects the top-mK per token, and shared experts always activate to absorb common cross-domain knowledge.

Model Variants: Pre-configured scales range from 1B to 1T parameters, with context windows up to 1 million tokens and output lengths up to 128k tokens.

Additional Hypotheses About the Full Mythos Architecture

Beyond the core looped transformer design, OpenMythos explores several additional hypotheses about what might constitute the complete Mythos architecture:

Loop Index Embedding: A RoPE-like embedding of the loop index injected at each step would allow the same parameters to implement different operations across iterations, substantially increasing expressiveness without parameter increase.

Adaptive Computation Time: The "overthinking problem"—where excessive recurrence degrades predictions—suggests Mythos likely includes a halting mechanism similar to the Adaptive Computation Time approach in Universal Transformers, allowing the model to dynamically decide when to stop looping.

Continuous Depth-wise Batching: Because all tokens share the same recurrent block, the model can exit the loop at different depths for different inputs, processing easy inputs quickly and hard inputs with more iterations within the same batch, potentially yielding 2-3x improvements in inference throughput.

Implications for the Future of AI

The OpenMythos project represents more than just an interesting technical exercise; it points toward a potential paradigm shift in AI architecture development. If Claude Mythos indeed operates on these principles, it suggests that future AI systems might:

1. Achieve deeper reasoning without proportional parameter increases
2. Develop systematic generalization capabilities that currently elude conventional architectures
3. Enable more efficient inference through adaptive computation depth
4. Balance breadth (via MoE) and depth (via looping) in novel ways

The project's documentation emphasizes that this is still theoretical: "OpenMythos is an independent, community-driven theoretical reconstruction based solely on publicly available research and speculation. It is not affiliated with, endorsed by, or connected to Anthropic or any of their proprietary systems."

Conclusion

OpenMythos provides a compelling framework for understanding what might power Claude Mythos, introducing the concept of Recurrent-Depth Transformers as a potential path toward more efficient and capable AI systems. The implementation demonstrates that looping mechanisms could enable deeper reasoning without proportional parameter scaling, potentially explaining Mythos's exceptional capabilities.

As the project continues to evolve and more research emerges about looped transformers, OpenMythos stands as an important contribution to the open-source AI community, offering researchers and developers a concrete implementation to build upon and test these novel architectural ideas. Whether or not it precisely captures Anthropic's proprietary design, the theoretical framework it presents represents a significant step toward more efficient and general AI systems.