High‑bandwidth memory (HBM) now consumes roughly two‑thirds of AI‑chip component spend, up from just over half a year ago. The shift reflects tighter memory supply, soaring prices, and design choices that favor larger, faster memory stacks. While investors see the trend as a cost‑driven pressure point, some designers argue that packaging innovations and software‑level optimizations could temper the growth.
Memory Takes the Lead in AI‑Chip Cost Structures
High‑bandwidth memory (HBM) has moved from 52 % to 63 % of total component spend for the major AI‑chip designers—Nvidia, AMD, Google, and Amazon—between Q1 2024 and Q4 2025. The data, compiled by Epoch AI, is weighted by production volume, so it reflects what actually ships rather than headline‑level forecasts.
Why the Shift Matters
- Absolute spend is exploding – HBM outlays grew from about $12 bn in 2024 to $32 bn in 2025, a $20 bn jump that dwarfs growth in any other component.
- Logic dies stay flat – The share for silicon dies hovered near 13 %, indicating that performance gains are being pursued more through memory bandwidth than through larger transistor counts.
- Packaging and aux components shrink – Advanced packaging fell from 19 % to 15 %, and auxiliary parts (interconnects, substrates, etc.) dropped from 15 % to 9 %. The decline is partly a mathematical effect of memory’s larger slice, but also reflects modest efficiency gains in CoWoS and substrate technologies.
- Total AI‑chip component spend doubled – From roughly $22 bn in 2024 to $52 bn in 2025, driven largely by HBM.
These numbers line up with public capex guidance from hyperscalers. Microsoft’s FY2026 budget now includes $25 bn earmarked for higher component prices, while Meta added $10 bn for the same reason. Both companies cite memory scarcity and price pressure as key drivers.
The Underlying Forces
1. Supply‑Side Tightness
HBM production is concentrated in a handful of fabs (TSMC, Samsung, and a few specialized lines). Capacity expansions take years, and the ramp‑up for 2025‑2026 is already booked. As demand from large‑scale training clusters outpaces supply, per‑gigabyte prices have risen 15‑20 % YoY.
2. Architectural Choices
Designers are stacking more memory per chip to keep data close to the compute cores, reducing latency and energy per operation. The trend toward larger HBM stacks (up to 48 GB per chip) directly inflates the bill‑of‑materials.
3. Economic Incentives
Customers are willing to pay a premium for higher throughput because it shortens training cycles and reduces overall cloud‑compute spend. That willingness translates into higher willingness to absorb HBM cost, at least for the top‑tier workloads.
Counter‑Perspectives
Packaging Innovation Could Re‑balance the Pie
Some engineers argue that the next wave of advanced packaging—chip‑on‑wafer‑on‑substrate (CoWoS 2.0) and heterogeneous integration with silicon‑photonic interposers—could lower the need for ever‑larger HBM stacks. By moving compute closer to memory through 3‑D integration, the same performance could be achieved with smaller memory footprints, pulling the memory share back down.
Software‑Level Optimizations
Frameworks such as TensorFlow XLA and PyTorch Dynamo are improving memory reuse and operator fusion, which can cut the required HBM bandwidth per training job. If these techniques become mainstream, the pressure to buy ever‑larger HBM may ease, even if the hardware cost curve stays steep.
Market Diversification
While Nvidia and AMD dominate the GPU market, Google’s TPU and Amazon’s Trainium are exploring HBM‑lite designs that rely more on on‑chip SRAM and lower‑cost DRAM. If these alternatives gain market share, the aggregate memory share could stabilize.
What to Watch in 2026
- Capacity announcements from Samsung and TSMC. A significant increase in HBM fab slots could temper price growth.
- Pricing trends for HBM‑3 vs. the upcoming HBM‑4. Early adopters may face a price premium that narrows as volumes rise.
- Packaging roadmaps from Intel and AMD. A successful rollout of next‑gen CoWoS could shift cost structures.
- Software‑level memory management breakthroughs in major AI frameworks, which could be tracked via GitHub activity and release notes.
Bottom Line
The data shows a clear re‑allocation of AI‑chip budgets toward memory, driven by supply constraints and architectural imperatives. While the headline numbers suggest an unstoppable march, the industry is already exploring pathways—advanced packaging, smarter software, and alternative architectures—that could reshape the cost mix. Observers should keep an eye on both the silicon supply chain and the software stack to gauge whether memory’s dominance will persist or give way to a more balanced component ecosystem.
All figures are based on Epoch AI’s publicly released analysis, licensed under CC‑BY. For the full methodology and raw data, see the Epoch AI report.
Comments
Please log in or register to join the discussion