The race to exascale computing isn't just about hardware. As systems like Lawrence Livermore National Laboratory's (LLNL) upcoming El Capitan supercomputer prepare to deliver unprecedented computational power, a less visible but equally critical challenge emerges: the crushing complexity of software dependency management. With these machines relying heavily on diverse GPU accelerators from NVIDIA, AMD, and Intel—each demanding unique programming environments—traditional manual integration approaches are reaching breaking point.

The Exascale Integration Burden

LLNL's flagship codes, like the Livermore Big Artificial Neural Network Toolkit (LBANN), exemplify the problem. LBANN relies on 70 external packages with 188 intricate dependency relationships. Maintaining compatibility across versions, compilers, build options, and OS updates for such stacks is a combinatorial nightmare.

"Software integration is a huge burden, and combinatorial HPC builds will catch up to us," warns LLNL computer scientist Todd Gamblin, leading the new BUILD project. "We don’t control all the software we’re trying to integrate with." Teams spend weeks resolving conflicts instead of innovating—a luxury exascale timelines won't allow.

Current solutions fall short:
1. Bundled Distributions (e.g., RedHat): Prioritize stability over currency, often using outdated versions.
2. Semantic Versioning: Relies on human accuracy and inconsistent adoption.
3. Live at Head: Forces universal latest-version usage, sacrificing needed flexibility.

Modeling Compatibility as an NP-Hard Puzzle

BUILD reframes dependency management as a logic problem akin to solving a jigsaw puzzle with evolving pieces. Each software update alters the "shape" (ABI - Application Binary Interface) of a package, potentially breaking connections. Finding compatible configurations across hundreds of packages is an NP-hard challenge—akin to solving complex edge-matching puzzles.

"The time is right to apply advanced solvers to the build configuration problem," asserts Gamblin. Modern conflict-driven clause learning (CDCL) solvers, successful in industrial applications, offer a path forward. BUILD aims to harness these to automate the discovery of valid, optimal software configurations.

The BUILD Blueprint: Four Thrusts for Automation

Funded as an LDRD Strategic Initiative, BUILD leverages Spack—LLNL's widely adopted open-source package manager (over 5,700 packages)—and attacks the problem on four fronts:

  1. Formal Compatibility Models: Developing machine-verifiable specifications for ABI (data types, functions) and critical properties (compilers, GPU runtimes).
  2. Automated Binary Analysis: Creating tools to extract ABI details directly from binaries, enabling pre-build compatibility checks without source code.
  3. Efficient Logic Solvers: Implementing solvers using full ABI compatibility data (not just version numbers) to find valid configurations within Spack's massive repository.
  4. ML-Driven Optimization: Training models on Spack's history to predict high-performance, correct configurations, guiding solvers toward optimal solutions.

Exascale and Beyond: The Impact

The potential payoff is massive: 10-100x acceleration in build and deployment workflows by enabling widespread binary reuse and eliminating manual conflict resolution. This directly benefits LLNL's mission codes for nuclear security and scientific discovery on El Capitan. Crucially, BUILD's approach transcends HPC:

  • Faster Porting: Simplifies adapting complex software stacks to new architectures.
  • Ecosystem Interop: Techniques could help Python packages and Linux distributions better manage low-level binary dependencies.
  • Developer Liberation: Frees researchers from integration drudgery to focus on core development and analysis.

"Our project’s multi-year design dovetails with the delivery of El Capitan," Gamblin notes. By embedding BUILD's solvers and analysis tools directly into Spack as open source, LLNL aims to fundamentally shift how high-performance software is integrated, making the unprecedented power of exascale systems truly accessible to developers pushing scientific frontiers. The solution to dependency hell may finally be within computational reach.