Vibecoding #2: A Practical Tool for Cloud Orchestration

The process of setting up a small, temporary cluster of machines for performance testing often feels more complex than it should be. For a project like TigerBeetle, where deterministic simulation is core to testing, verifying performance on real hardware in a cloud environment remains a necessary step. The workflow typically involves procuring several machines, deploying custom binaries, and coordinating load testing across them—a task that, despite decades of cloud computing, still requires significant manual effort.

This article documents the creation of a personal tool, box, designed to streamline this process. The goal was to spin up a fleet of ephemeral machines on demand, sync local code to them, and run commands in parallel, while still allowing for individual SSH access when needed. The solution draws inspiration from three sources: the rsyscall project's philosophy of direct-style distributed programming, Peter Bourgon's remote development scripts that synchronize state across machines, and the dax library for shell scripting in JavaScript. The core insight is that managing multiple machines should feel like managing a single logical environment, with the ability to choose where commands execute.

The implementation was undertaken with the assistance of AI, specifically ChatGPT for initial specification and Claude for code generation. The author, self-described as being at the "tail end of AI adoption," sought to test the hypothesis that AI is particularly good at wrangling the baroque APIs of modern cloud providers like AWS. The process began with a high-level description of the desired functionality, which was expanded by ChatGPT into a more detailed specification. This phase surfaced practical considerations like machine identification (leading to a sequential 0,1,2 naming scheme), permissions, and network rules.

A key finding was that for the current generation of AI, examples are far more valuable than abstract rules. The initial specification failed to correctly capture the author's intent for directory mapping—materializing a local directory structure on remote machines with different home paths. A concrete example proved more effective for clarification than prose. Similarly, the desire for auto-shutdown of idle machines was refined through a back-and-forth with the AI, ultimately leading to a simpler, time-based solution rather than a complex idle-detection system.

The author's experience diverged significantly between a "one-shot" generation attempt and an incremental, skeleton-driven approach. The initial one-shot result, while potentially functional, lacked a coherent structure and "character." It felt like generic code, difficult to understand or maintain. This aligns with the author's development philosophy: solutions are discovered through refactoring and working with code, not designed up-front. A one-shot dump of code prevents this iterative discovery process.

The incremental approach proved far more successful. The author began by manually writing a TypeScript type definition for the CLI commands and a skeleton main function. This provided a structural framework that the AI could then fill in. For instance, the CLIParse function was completed by Claude based on the provided types and a fatal error handler. This pattern—providing the structure and letting the AI handle the details—was repeated throughout the project. The author defined the control flow in main functions (e.g., mainDestroy) which looped over boxes and called instance-level functions (e.g., instanceDestroy). When implementing a new subcommand, the author would write the skeleton for both, and then ask the AI to implement the instance-level logic according to the specification.

The AI was particularly effective at handling the specific, non-reusable knowledge required for AWS API calls. For example, generating the correct aws ec2 run-instances command with spot instance options, tag specifications, and user data is a task that involves remembering exact flag names and JSON structures—a perfect use case for AI assistance. The author notes that they would have spent hours on this via trial and error, a process the AI could accelerate.

However, the collaboration wasn't flawless. The AI sometimes missed the overall architectural pattern (like the main/instance split) and required course correction. It also made naive bugs, such as using aws ec2 wait instance-running instead of aws ec2 wait instance-status-ok, a subtle but critical difference. Debugging these issues involved pasting errors back to the AI, which was remarkably effective. For instance, when a disk mount broke SSH access, the AI immediately identified the cause: mounting over /home overwrote the SSH keys. This highlights a significant value of AI in development: it can quickly point to relevant logs and explain obscure error messages, saving the developer from the common time sink of searching for diagnostic information.

The final tool, box, works as intended, allowing the author to create, sync to, and run commands on a cluster of machines with simple commands like box create 3, box sync 0,1,2, and box run 0,1,2 ./zig/zig build. The codebase, while not perfect, is maintainable because it reflects the author's structural choices. The experience underscores that AI is a powerful tool for filling in blanks and handling API minutiae, but it benefits from human guidance on architecture and design philosophy. The process of building a tool to solve a personal pain point, using AI to navigate complex cloud APIs, resulted in a practical utility and a deeper understanding of effective human-AI collaboration in software development.