The Hidden Runtime: Why Your LLM Behaves Differently in Production

In the world of large language models, we often think of deployment as simply loading weights onto hardware. But as Hazem Ali, Microsoft AI MVP and distinguished AI architect, demonstrates in his comprehensive analysis "The Hidden Architecture of Nano Architectures," this mental model fundamentally misunderstands production AI systems.

What Changed: From Model to Runtime

The core revelation is that in production environments, you don't deploy just a model—you deploy a runtime that selects execution plans under constraints. This distinction explains why the same prompt, checkpoint, and temperature setting can produce different outputs only when the system is under real load.

"Weights are static. Behavior is a property of the executed plan," Ali explains. The executed plan depends on system state, which changes under load. This understanding shifts how we approach cloud infrastructure for AI workloads.

Provider Comparison: Cloud Platforms and Execution Regimes

Different cloud providers offer varying capabilities for managing these execution regimes, which directly impacts LLM behavior consistency:

Azure

Microsoft has positioned Azure as an enterprise-grade platform for AI workloads with strong emphasis on reproducibility and governance.

• Azure ND H100 v5: Explicitly designed for tightly coupled scale-up and scale-out generative AI workloads, with fabric characteristics that influence execution regimes. Azure's documentation highlights how the fabric matters, not just GPUs.
• Confidential Computing: Azure offers confidential VMs with H100 support, which provides stronger isolation but can restrict sharing and change execution regimes. Azure Confidential VMs
• Managed Services: Azure AI Studio and Azure Machine Learning provide managed serving infrastructure that abstracts some runtime complexities but still exposes regime-dependent behavior.

AWS

AWS offers a range of GPU instances and services that influence execution regimes differently:

• Trainium and Inferentia: AWS's custom silicon changes the execution equation entirely, with different memory hierarchies and tensor core architectures that create distinct regime behaviors. AWS Trainium Documentation
• EC2 P4d/P5 instances: NVIDIA A100 and H100 instances that offer different networking topologies affecting multi-node tensor parallel behavior. EC2 GPU Instances
• SageMaker: Provides managed endpoints but with different underlying infrastructure that may produce different regime transitions compared to self-hosted solutions.

Google Cloud

Google's approach emphasizes TPUs and integrates with their broader data ecosystem:

• TPU Pods: Google's tensor processing units offer a fundamentally different execution model compared to GPUs, with different memory hierarchies and parallel execution patterns. Google Cloud TPUs
• Vertex AI: Google's managed AI platform provides consistent environments but may abstract away regime-dependent behaviors that surface in self-managed deployments.

Multi-Cloud Considerations

When designing multi-cloud strategies for LLM inference, understanding these execution regimes becomes critical:

1. Reproducibility Challenges: Different cloud providers have different default behaviors for kernel selection, memory management, and batching that can lead to different outputs even with identical models and prompts.

2. Cost-Performance Tradeoffs: Each cloud provider's infrastructure creates different regime boundaries. What appears as a "performance optimization" in one cloud may actually push the system into an unstable regime in another.

3. Migration Complexity: Moving LLM workloads between clouds requires more than just model checkpoint transfer. It requires understanding and potentially reconfiguring the runtime behavior to match the new execution regime.

Business Impact: Regime-Dependent Behavior and Enterprise AI

The implications of this understanding extend far beyond technical curiosity into enterprise AI strategy and operational readiness:

The p95/p99 Problem

Ali observes that behavior drift typically surfaces at p95 and p99 latency levels—precisely where it hurts most in production. This creates a significant operational challenge:

• Incident Response: Teams struggle to explain why "identical" requests produce different outputs, often misattributing the issue to model randomness rather than execution regime changes.
• Audit and Compliance: In regulated industries, inconsistent outputs can create compliance challenges, especially when systems behave differently under load during critical operations.
• Customer Trust: When AI systems produce inconsistent responses, it undermines user confidence in the technology.

Infrastructure as Behavior Control

Cloud infrastructure choices directly influence which execution regimes your system will enter under pressure:

• Instance Selection: The choice between GPU types (A100 vs. H100), memory configurations, and networking topologies affects regime boundaries.
• Isolation vs. Performance: Stronger isolation (required for multi-tenant deployments) can restrict sharing and push the system into different execution regimes.
• Observability: The quality of telemetry available for tracking execution regimes varies significantly between cloud providers.

Operational Recommendations

Based on Ali's analysis, organizations should:

1. Log Execution Contracts, Not Just Prompts: Track the effective request after shaping, memory headroom, and execution state—not just the raw user input.

2. Test Under Regime Conditions: Validate behavior not just at idle, but under sustained concurrency, mixed sequence lengths, and realistic memory pressure.

3. Measure Early Logit Margins: Track the difference between top candidate logits in early decoding steps as a stability budget.

4. Choose Infrastructure Based on Regime Requirements: Select cloud providers and instance types that maintain stable execution regimes under your expected load patterns.

Migration Strategy: From Demo to Production Mental Models

The article highlights a critical mental model shift required for successful LLM deployments:

Demo Equation: y = f(x, θ)

• One prompt in, one checkpoint, one output
• Assumes deterministic behavior regardless of system state

Production Equation: y = Decode(Exec(θ, x; s))

• Weights (θ) remain constant, but executed plan (Exec) depends on system state (s)
• Behavior emerges from the interaction between model, input, and runtime constraints

When migrating LLM workloads from development to production or between cloud providers, organizations must account for this shift. The same model can behave differently across environments due to differing runtime behaviors and regime boundaries.

Conclusion: Infrastructure as Part of the Model

As Ali states, "You did not deploy weights. You deployed a physics constrained runtime that contains weights." This understanding transforms how we approach cloud infrastructure for AI workloads.

Cloud providers are no longer just delivery mechanisms for compute—they are active participants in shaping model behavior through their execution regimes. Organizations that recognize this can design more reliable, predictable AI systems. Those that continue to operate with the "demo mental model" will continue to be surprised by production behavior.

The future of enterprise AI requires a deeper integration of infrastructure understanding with model deployment strategies—one that acknowledges the complex interplay between weights, runtime, and system state that ultimately determines what users see.

For those operating at scale, the message is clear: to build reliable AI systems, you must understand and control the execution regimes that shape your model's behavior in production.