Caching Beyond Compilers: Optimizing Firefox Builds with Lua Plugins

The recent optimization to Firefox builds, achieving a 17% performance improvement through WebIDL code generation caching, represents a significant advancement in build system architecture. This improvement, while seemingly modest in isolation, demonstrates a fundamental shift in how we approach caching beyond traditional compiler invocations, opening new possibilities for accelerating complex software compilation pipelines.

At the core of this optimization lies the WebIDL binding code generation step—a critical yet often overlooked component of the Firefox build process. When building Firefox, the system runs python3 -m mozbuild.action.webidl to transform hundreds of .webidl files into thousands of C++ binding files including headers, implementations, and declarations. This step, while not excessively slow in isolation, runs on every clobber build and produces entirely deterministic output given the same inputs, making it an ideal candidate for caching. The challenge historically was that compiler caches like ccache and sccache were designed specifically for compiler invocations, not Python-based code generation tools.

The solution implemented in Bug 2027655 demonstrates architectural elegance in its simplicity. By modifying dom/bindings/Makefile.in to conditionally pass $(CCACHE) as a command wrapper to the py_action call, the build system now intercepts the Python codegen process through buildcache's Lua plugin system. This conditional compilation approach ensures compatibility while enabling the new functionality only when buildcache is in use. The introduction of the fourth argument to the py_action macro represents a subtle but significant enhancement to Mozilla's build infrastructure.

What makes this approach particularly compelling is the implementation of webidl.lua, a custom Lua wrapper that teaches buildcache how to handle this non-traditional build step. This wrapper must answer three fundamental questions: whether it can handle a given command (by matching mozbuild.action.webidl in the arguments), what the inputs are (all .webidl files and Python scripts), and what the outputs are (the generated binding files and state files). This metadata allows buildcache to create proper input hashes, check for cached results, and either replay cached outputs or execute the command and store the results.

The performance benchmarks reveal the tangible impact of this optimization:

• No cache: 5m35s
• ccache warm: 3m21s
• sccache warm: 2m49s
• buildcache warm: 1m27s
• buildcache warm with plugin: 1m12s

These numbers demonstrate that buildcache with the Lua plugin not only outperforms other caching solutions but achieves additional optimization through specialized handling of the WebIDL step. The 15-second improvement from the Lua plugin may seem incremental, but it represents a 17% performance gain over an already optimized buildcache implementation—a meaningful improvement for developers working frequently with Firefox builds.

The broader implications of this approach extend far beyond the WebIDL optimization. The Lua plugin system enables caching of any deterministic build step with known inputs and outputs, opening possibilities for optimizing other codegen actions in the Firefox build process. This represents a paradigm shift in build system design, moving from compiler-specific caching to a more holistic approach that can handle the diverse tooling used in modern software development.

Setting up this optimization requires additional configuration beyond standard buildcache usage. Developers must clone the buildcache-wrappers repository and either modify their ~/.buildcache/config.json file or set the BUILDCACHE_LUA_PATH environment variable in their mozconfig. The need for a larger max_local_entry_size (2.5 GB) also highlights the diverse caching requirements across different components of the Firefox build system, particularly for Rust crates that produce substantial cache entries.

While the performance gains are significant, several considerations temper the enthusiasm. First, these benefits primarily affect clobber builds with warm caches; regular incremental builds may see less dramatic improvements. Second, the benchmarks represent single runs on one machine rather than rigorous scientific testing. Third, the optimization is specific to buildcache, which lacks the widespread adoption of more established tools like ccache.

Looking forward, the Firefox build system could benefit from applying this Lua plugin approach to other codegen steps, potentially compounding performance improvements. The success of the WebIDL wrapper serves as both a proof of concept and a template for future optimizations, demonstrating that even mature build systems can benefit from architectural innovation.

For developers interested in implementing this optimization, the necessary changes are available in the central repository, with detailed setup instructions provided in the original article. The buildcache-wrappers repository contains the Lua wrapper implementation that makes this optimization possible.

This optimization ultimately represents more than just a performance improvement; it exemplifies how thoughtful build system design can significantly enhance developer productivity, reducing friction in the edit-compile-test cycle that forms the core of modern software development.