No, Really, Bash Is Not Enough: Why Large-Scale CI Needs an Orchestrator

{{IMAGE:1}}

The Infrastructure Tipping Point

When CI pipelines evolve from running simple test suites to becoming mission-critical infrastructure supporting hundreds of engineers, the limitations of bash scripting emerge with striking clarity. Teams experiencing CI failures that cascade into deployment delays, PR backlogs, and missed product deadlines face problems fundamentally different from those solved by sequential shell commands. The core contention isn't that bash lacks utility—it excels at linear workflows—but that distributed build systems require architectural solutions beyond scripting languages.

Formal Foundations of Failure

Research from the paper "Build Systems à la Carte" (Mokhov, Mitchell, Peyton Jones) provides rigorous framework for understanding CI systems. Their taxonomy reveals bash-based approaches occupy the "busy" cell—rebuilding everything without dependency tracking or incremental computation. This contrasts sharply with mature systems like Bazel (constructive traces) or Shake (suspending scheduler) that minimize redundant work through persistent build information. Bash lacks mechanisms for:

• Dynamic dependency resolution
• Cloud caching architectures
• Early cutoff optimization
• Minimal rebuild guarantees

Attempting to implement these patterns in bash means manually constructing distributed systems using tools designed for command sequencing, inevitably resulting in race conditions and unpredictable failures.

The Combinatorial Sprawl Challenge

Modern CI requirements quickly overwhelm scripting approaches. Consider a mid-sized organization's typical PR pipeline:

1. Parallelized test suites (backend/frontend/integration)
2. Multi-architecture Docker builds
3. Code coverage enforcement
4. Infrastructure-as-code validation
5. Performance benchmarking
6. Security scanning

Expressing dependencies between these steps—where integration tests require completed Docker builds, or Terraform checks trigger only on infra changes—demands a directed acyclic graph (DAG) structure. Bash implementations devolve into:

• Fragile file-based locking systems
• Nested background processes with & and wait
• Manual state management via shared filesystems
• Undocumented ordering constraints

The resulting 1,000-line scripts become unmaintainable entanglement points where failures manifest as "spectral contamination"—unreproducible errors caused by residual state from concurrent processes.

Isolation as Non-Negotiable

Shared execution environments create systemic fragility:

Resource Collisions

• Port conflicts (multiple tests claiming port 5432)
• Filesystem corruption (concurrent npm installs)
• Docker cache corruption (documented BuildKit issues)
• OOM killer terminating unrelated processes

Environmental Contamination Third-party tools rarely consider concurrent execution:

• Docker's cache locking failures under parallel builds
• npm's vulnerability to EINTEGRITY errors under concurrency
• Yarn's requirement for explicit mutex flags

Proper orchestrators solve this by provisioning isolated environments per job—each with dedicated filesystems, networks, and resources—eliminating cross-build interference without requiring hermetic behavior from every tool.

The Orchestrator Imperative

Purpose-built CI systems provide essential capabilities absent in scripts:

Control Plane Functions

• Fleet-wide observability dashboards
• Resource-aware scheduling
• Graceful node draining
• Historical performance analytics
• Pipeline-aware retry mechanisms

Architectural Safeguards

• Cgroup-enforced memory limits (preventing OOM domino effects)
• Dependency validation (blocking invalid artifact references)
• Hermetic execution environments
• First-class artifact management

These features represent institutional knowledge encoded into tools—lessons learned from scaling failures that scripting approaches inevitably rediscover through painful experience.

Addressing Counter-Perspectives

"Makefiles suffice for complexity" While Make occupies a valid point in the build system taxonomy (topological scheduler + dirty bits), it lacks support for dynamic dependencies, cloud caching, or hermetic execution—capabilities essential at scale.

"LLMs can generate advanced bash" Large language models excel at syntactic solutions but cannot encode operational wisdom. An LLM might write cgroup configurations but won't anticipate that namespace isolation proves insufficient for GPU contention, or that Docker cache collisions require VM-level separation.

"Orchestrators add unnecessary complexity" The simplicity of bash shifts complexity into operational burden. Teams eventually pay cognitive debt through:

• Midnight debugging of concurrent failure modes
• Custom tooling for observability
• Manual resource contention resolution

The Path Forward

For organizations experiencing CI-induced productivity erosion, solutions exist across the orchestration spectrum:

• Agent-based systems (Buildkite, GitLab CI)
• Pipeline-native tools (Tekton, Concourse)
• Cloud-native frameworks (Dagger)

Common traits distinguish these from scripting:

1. Explicit dependency modeling via DAGs
2. Built-in artifact passing
3. Resource isolation guarantees
4. Operational control planes

The transition acknowledges CI as inherently distributed computation—a domain where ad-hoc scripting eventually yields to engineered systems. Teams crossing this threshold discover orchestrators aren't overhead but foundational infrastructure, as indispensable as databases for state management or load balancers for traffic distribution.