Memory Bandwidth: The Unseen Barrier to Local LLM Performance on AMD Hardware

The performance of local large language models on consumer hardware remains a puzzle wrapped in an enigma, with many enthusiasts focusing on computational capabilities while overlooking the critical role of memory architecture. A detailed benchmarking investigation on a Framework 13 laptop equipped with AMD's Ryzen AI 9 HX 370 processor and Radeon 890M integrated graphics reveals that the true limitation isn't processing power but memory bandwidth—a finding that fundamentally changes how we approach local LLM optimization.

The Hardware Reality

The Framework 13 AMD Strix Point configuration presents an interesting case study in modern APU design:

• CPU: 12-core, 24-thread AMD Ryzen AI 9 HX 370 with Radeon 890M iGPU
• Memory: 64GB DDR5 (2x32GB SO-DIMMs, upgradeable to 96GB)
• Memory Interface: 128-bit bus with DDR5-5600 (theoretical max bandwidth: 89.6 GB/s)
• GPU Architecture: 16 Compute Units, RDNA 3.5

What emerges from the detailed system analysis is that the memory subsystem—specifically the 128-bit memory bus—represents the hard ceiling for performance, regardless of software optimizations or driver improvements. As the author notes, "The 128-bit DDR5-5600 bus at 89.6 GB/s is the hard wall."

The Bandwidth Bottleneck

For text generation tasks—the primary user-facing aspect of LLM interaction—performance is almost entirely determined by memory bandwidth. Each generated token requires reading most of the model's weights from memory, making the GPU wait for data rather than being compute-bound.

Benchmark results with Qwen3-8B Q4_K_M demonstrate this principle clearly:

• Power-saver mode (battery): 9.87 tokens/second (55% memory bandwidth utilization)
• Performance mode (AC): 13.41 tokens/second (75% memory bandwidth utilization)

The difference between these modes isn't merely software-related—it's the memory controller and GPU operating at higher clock speeds when connected to AC power. This explains why proper benchmarking methodology must account for power states, as testing on battery measures the power governor's behavior rather than pure hardware capability.

Software Pathways: Vulkan vs ROCm vs CPU

The Framework 13 offers three distinct software pathways for LLM inference, each with different performance characteristics:

1. Vulkan (RADV driver): Graphics API repurposed for compute, integrated with the system's display driver
2. ROCm/HIP: AMD's CUDA alternative, optimized purely for compute workloads
3. CPU: Direct CPU processing using AVX-512 instructions on Zen 5

Contrary to expectations, Vulkan significantly outperformed ROCm for interactive text generation (13.41 vs 4.76 tokens/second), despite ROCm being 41% faster at prompt processing. This discrepancy stems from how these APIs interact with the memory subsystem:

• The GPU accesses memory through a direct wide path to the memory controller
• The CPU accesses memory through the Infinity Fabric, which provides roughly half the bandwidth
• ROCm, while efficient for certain workloads, cannot overcome the fundamental memory bandwidth limitations

The author's conclusion is pragmatic: "Since tg is the user-facing latency, Vulkan wins for interactive use. ROCm is worth it for RAG/long-context workloads where prompt processing dominates."

The Speculative Decoding Breakthrough

While the memory bandwidth wall represents a fundamental hardware limitation, the investigation uncovered a software technique that dramatically improves performance: speculative decoding. This approach uses a small "draft" model to generate candidate tokens quickly, then the larger model verifies them all in a single batch operation.

The results are transformative:

• Qwen3-8B with Qwen3-0.6B draft (draft-max=4): 21.2 tokens/second (+64%)
• Qwen3-8B with Qwen3-0.6B draft (draft-max=8): 22.0 tokens/second (+71%)
• Qwen3-8B with Qwen3-0.6B draft (draft-max=16): 22.9 tokens/second (+78%)
• Qwen3-8B with Qwen3-0.6B draft (draft-max=32): 23.5 tokens/second (+82%)

This technique works around the memory bandwidth limitation by amortizing the cost—instead of reading the full model weights per token, the system reads them once and verifies multiple tokens in batch. The asymmetry between prompt processing (24x faster than token generation on this hardware) and text generation makes this approach particularly effective.

Interestingly, the size ratio between draft and target models proves critical. The 0.6B-to-8B model ratio (13:1) represents the sweet spot, with smaller ratios like 3B-to-7B providing more modest gains (+36%). This suggests that future optimizations should focus on maximizing this asymmetry ratio.

Platform Comparison: AMD vs Apple Silicon

A comparison with Apple Silicon systems reveals how memory architecture fundamentally influences LLM performance:

• Framework 13 (DDR5-5600, 128-bit): 89.6 GB/s bandwidth, ~13 tokens/second for 8B model
• MacBook Pro 16" M1 Pro (LPDDR5, 256-bit): 200 GB/s bandwidth, ~22 tokens/second
• MacBook Pro 16" M3 Pro (LPDDR5, 192-bit): 150 GB/s bandwidth, ~16 tokens/second
• MacBook Pro 16" M3 Max (LPDDR5, 512-bit): 400 GB/s bandwidth, ~45 tokens/second

Apple's advantage stems from wider memory buses and unified memory architecture that eliminates the Infinity Fabric bottleneck. However, the Framework's upgradeable SO-DIMMs offer a different advantage—support for up to 96GB of RAM, enabling models that wouldn't fit on Apple's more limited configurations.

Notably, the comparison reveals a bandwidth regression in Apple's newer M3 Pro, which narrowed its memory bus from 256-bit to 192-bit compared to the M1 Pro, demonstrating that newer hardware doesn't always mean better performance for specific workloads.

Practical Implications and Future Directions

The investigation yields several practical insights for local LLM deployment:

1. Power profiles matter more than generally acknowledged—switching from power-saver to performance mode increased token generation by 36%
2. Vulkan remains the optimal choice for interactive workloads on AMD hardware despite ROCm's advantages for certain tasks
3. Speculative decoding offers the largest performance gains without requiring hardware upgrades
4. Memory bandwidth utilization should be the primary metric for evaluating LLM performance

Looking forward, the author suggests several promising avenues:

• Testing Qwen3-14B with Qwen3-0.6B draft, which could represent the practical sweet spot for this hardware
• Benchmarking newer models like Qwen3-Coder-30B-A3B and Qwen2.5-Coder-14B
• Revisiting the coding benchmarks from Part 1 with speculative decoding enabled

The collaborative nature of this investigation—between human researcher and AI assistant—highlights how technical analysis benefits from both domain expertise and computational power. The iterative process of challenging claims, verifying data, and refining explanations produced insights that neither approach might have achieved independently.

For those interested in reproducing these results, the article provides comprehensive commands for hardware detection, benchmarking different backends, and calculating memory bandwidth utilization. This transparency enables the broader community to verify findings and build upon this foundation.

As local LLM capabilities continue to evolve, understanding these fundamental hardware limitations becomes increasingly important. While speculative decoding offers a powerful workaround, the memory bandwidth ceiling remains a constraint that only architectural improvements can overcome—a reality that will shape the future of on-device AI capabilities.