Datadog's Binary Optimization Journey: Reducing Go Binaries by 77% Without Sacrificing Features

The evolution of software often comes with an unintended consequence: growth. For Datadog, this manifested as a near threefold increase in their Agent's binary size over five years, from 428 MiB to 1.22 GiB. This expansion, while reflecting added capabilities and integrations, created significant constraints in resource-sensitive environments like serverless platforms, IoT devices, and containerized workloads. Rather than accepting this trajectory as inevitable, Datadog engineers embarked on an ambitious optimization journey that would ultimately reduce their Go binaries by up to 77% without removing a single feature.

The Challenge: Understanding Binary Growth

The Datadog Agent presents a complex architectural puzzle. While appearing as a unified product to users, it comprises dozens of platform-specific builds tailored to diverse environments including Docker, Kubernetes, Heroku, and various Linux distributions. These builds share a common codebase but incorporate different features through extensive use of Go build tags and dependency injection.

This architectural complexity contributed to the binary size growth observed between versions 7.16.0 and 7.60.0. The compressed size of the Linux amd64 deb package more than doubled, from 126MiB to 265MiB, while the uncompressed size jumped from 428MiB to 1248MiB—a staggering 192% increase. This growth impacted network costs, resource usage, and the Agent's usability in constrained environments.

Systematic Dependency Auditing: The Foundation of Optimization

The first major pillar of Datadog's optimization strategy involved a comprehensive audit of dependencies. The Go compiler's approach to dependency inclusion is more nuanced than it might initially appear. It works at the package level, compiling each required package into an intermediate artifact that the linker later combines into a final binary, retaining only the reachable symbols.

To understand which dependencies were actually being included, Datadog engineers leveraged several tools:

1. go list: Used to display all packages included when building a binary for specific OS, architecture, and build tags.
2. goda: A tool that creates interactive graphs of package imports, showing both direct and indirect dependencies.
3. go-size-analyzer: Displays the size contribution of each dependency in a Go binary, either as text or in an interactive web interface.

These tools revealed a critical insight: dependencies were often included unintentionally through import chains, even when their functionality wasn't actually being used.

A 36 MiB Victory: One Function, Hundreds of Removed Packages

The most dramatic example of dependency optimization occurred in the trace-agent binary. Despite updating the tagging logic to remove Kubernetes dependencies, go list still showed 526 packages from k8s.io being included, accounting for at least 30 MB of binary size.

Using goda, engineers traced the import path back to a single package in their codebase. This package was only included for one function that didn't actually depend on any Kubernetes code. By moving this function into its own package and updating the relevant imports, the compiler was able to trim all the unused dependencies.

The result was remarkable: 570 packages removed from the Linux trace-agent binary and a size reduction of approximately 36 MiB—more than half of the binary's size. This extreme example wasn't unique; similar cases, though with smaller impacts, were found throughout the codebase. Through systematic listing, auditing, and pruning of these imports, Datadog significantly reduced binary sizes across the Agent.

Enabling Linker Optimizations: The Method Dead Code Elimination

While dependency pruning provided substantial gains, Datadog engineers discovered another opportunity: enabling a powerful but often overlooked linker optimization known as "method dead code elimination." This optimization removes unused methods when using the reflect package, but is disabled by default in many scenarios.

The challenge stems from how the reflect package works. When using MethodByName with a non-constant method name, the linker cannot know at build time which methods will be used at runtime. Consequently, it must keep every exported method of every reachable type, along with all their dependencies, which can drastically increase binary size.

To identify and fix these issues, Datadog developed and used the whydeadcode tool, which analyzes linker output to determine why the optimization is disabled and identifies the specific call chains causing the problem. Through this process, they discovered that only about a dozen dependencies needed patching to exclude problematic uses of reflect.

For dependencies like text/template and html/template from the standard library, which didn't have immediate fixes available, Datadog forked these packages into their codebase and patched them to disable method calls in their own template usage. As a result, they achieved an average 20% binary size reduction across all binaries, ranging from 16% to 25% depending on the specific binary, compounding to a total reduction of around 100 MiB.

Notably, these contributions benefited the wider Go community. Kubernetes project contributors began enabling the same optimization and reported 16% to 37% size reductions, demonstrating how Datadog's work had ripple effects beyond their own codebase.

Uncovering Architecture-Specific Linker Behaviors

During their optimization efforts, Datadog engineers encountered a puzzling discrepancy: while the method dead code elimination was significantly reducing binary sizes on arm64 architecture, it had minimal impact on amd64 builds.

Investigation revealed an unexpected behavior in the Go linker. Simply importing the plugin package from the standard library causes the linker to treat the binary as dynamically linked, which disables method dead code elimination and forces the linker to keep all unexported methods. This explained the architecture-specific difference, as the plugin package was imported in amd64 builds but not arm64 ones.

Using goda to trace the import, engineers found that the containerd package was responsible for importing the plugin package—a feature that the Datadog Agent didn't actually rely on. Rather than simply removing the import, they collaborated with the containerd project to add build tags that would allow selective inclusion without breaking existing users.

This specific change resulted in a 245 MiB size reduction for Datadog's main Linux amd64 artifacts—roughly a 20% decrease in total size at the time—benefiting approximately 75% of their users.

Results: A Return to Leaner Binaries

The cumulative effect of these optimization strategies was dramatic. Between versions 7.60.0 (December 2024) and 7.68.0 (July 2025), Datadog reduced their binary sizes across the board:

• Core Agent: 236 MiB → 103 MiB (decrease of 56%)
• Process Agent: 128 MiB → 34 MiB (decrease of 74%)
• Trace Agent: 90 MiB → 23 MiB (decrease of 74%)
• Security Agent: 152 MiB → 35 MiB (decrease of 77%)
• System Probe: 180 MiB → 54 MiB (decrease of 70%)

Overall, the compressed and uncompressed sizes of their .deb package dropped by about 44%, from 265 MiB to 149 MiB compressed, and from 1.22 GiB to 688 MiB uncompressed. Perhaps most importantly, these reductions were achieved without removing any functionality. Every capability added over the previous five years remained intact, while the binaries became smaller, cleaner, faster to distribute, and more memory-efficient.

Implications for the Go Ecosystem

Datadog's optimization journey offers valuable insights for other organizations working with Go:

1. Dependency Auditing is Essential: Even in well-structured codebases, dependencies can be unintentionally included through import chains. Systematic auditing using tools like goda and go-size-analyzer can reveal significant optimization opportunities.

2. Linker Optimizations Matter: The Go linker includes powerful optimizations like method dead code elimination that are often overlooked. Understanding and enabling these can yield substantial size reductions.

3. Community Collaboration Pays Dividends: By working with upstream projects like containerd and Kubernetes, Datadog not only solved their own problems but also contributed to improvements that benefited the wider ecosystem.

4. Architecture-Specific Behaviors Can Surprise: The Go compiler and linker can exhibit different behaviors across architectures, making comprehensive testing across target platforms crucial.

Looking Forward

Datadog's binary optimization journey demonstrates that feature growth and size efficiency need not be mutually exclusive. Through systematic engineering practices and deep understanding of the Go toolchain, they've created a more efficient implementation of their Agent while maintaining all existing functionality.

For organizations facing similar challenges with binary bloat, Datadog's experience offers a roadmap: start with dependency auditing, explore linker optimizations, and be prepared to engage with upstream projects when necessary. The result is not just smaller binaries, but a more efficient, maintainable, and deployable software product that can thrive in increasingly diverse computing environments.

For those interested in exploring these optimization techniques further, the tools mentioned in this article—goda, go-size-analyzer, and whydeadcode—provide valuable starting points for analyzing and improving Go binary sizes. As the Go ecosystem continues to evolve, such optimization practices will become increasingly important for building efficient, scalable software.