When stdout is the only door: hacking file collection in production environments

The story begins with a seemingly simple task: running a command on production boxes that generates CSV reports for record-keeping. The initial approach worked fine for a single box—log in, run the command, manually copy the CSV files using kubectl cp, and call it a day. But when the scope expanded to 100 boxes, the limitations of the existing tooling became painfully apparent.

The internal management tool was designed for one thing: running commands across multiple boxes with proper concurrency control and log collection. It would happily execute my_command across all matching boxes and save the stdout logs as box-1.log, box-2.log, and so on. But it had no mechanism for collecting files that the command might create during execution. The tool's philosophy was simple: run the command, capture the output, shut down. No file collection, no passing go, no collecting $200.

The command itself was solid—properly reviewed, checked into the repository, and doing exactly what it needed to do. The problem was that it wrote to disk, creating CSV files that needed to be collected for record-keeping purposes. The "right" solution would have been to rewrite the command to upload files to S3 or some other storage system, but that would involve going through the entire review process and deployment cycle. For a one-off task that might never be needed again, this felt like overkill.

This is where operational thinking diverged from software engineering thinking. Instead of asking "how should this be done properly?" the question became "how can I get this done with the tools I have?" The answer lay in understanding that stdout is the universal interface. Everything the management tool collected was going through stdout, so if the CSV files could be funneled through stdout, they would be automatically collected.

The solution was elegantly simple: modify the command invocation to echo markers between the CSV contents. By running my_command followed by echo "YYY 1", cat widgets.csv, echo "YYY 2", cat thingies.csv, and so on, the CSV data would flow through stdout and be captured in the log files alongside the regular command output. The markers (YYY 1, YYY 2, etc.) served as delimiters that could be used to split the log files back into their constituent CSV files.

The Python processing script that followed was straightforward. It iterated through each log file, split the contents using the YYY markers as delimiters, and wrote the resulting CSV data to appropriately named files in a results directory. The final YYY 4 marker served as a trailer to ensure all expected output had been captured. The entire solution—command modification plus processing script—came in at around 20 lines of code.

This approach exemplifies a crucial mindset shift that often separates effective operators from frustrated engineers: when faced with a tool that almost does what you need, look for ways to adapt your workflow to the tool rather than immediately reaching for a rewrite. The management tool wasn't broken—it was doing exactly what it was designed to do. The requirement had simply evolved beyond its original scope.

The elegance of this solution lies in its pragmatism. It avoided the overhead of code review, deployment, and potential production impact that would have come with modifying the original command. It leveraged existing infrastructure (the log collection system) rather than building new systems. And it solved the immediate problem without creating technical debt or maintenance burden.

This kind of thinking—acting like an operator when doing operations—is often undervalued in software development cultures that emphasize "doing things right" over "getting things done." There's a time and place for proper architectural solutions, but there's also immense value in being able to improvise effective solutions with the tools at hand. The best operators I've worked with share this ability to see past the intended use of a tool to its actual capabilities.

The broader lesson extends beyond this specific scenario. In complex distributed systems, the interfaces between components often become the most important constraints. When stdout is your only door, learning to fit everything through that door—even when it seems too small—is a valuable skill. Sometimes the hammer and nail metaphor applies in reverse: when you have a hammer, everything looks like a nail, but sometimes you need to realize that your hammer can also be used as a doorstop, a paperweight, or a CSV file transfer mechanism.

This experience also highlights the importance of designing tools with extensibility in mind. The management tool's single-purpose design made it simple and reliable for its core use case, but it also made it brittle when requirements changed. A more flexible design might have included hooks for file collection or other post-processing steps. However, building for every possible future requirement can lead to over-engineered solutions that are complex and difficult to maintain.

The tension between simplicity and flexibility is a constant challenge in tool design. The management tool chose simplicity and succeeded at its primary purpose. The operator chose flexibility and succeeded at the immediate task. Both approaches have merit, and understanding when to apply each is part of the art of building and operating software systems.

In the end, the solution worked perfectly. All 100 boxes were processed, all CSV files were collected, and the record-keeping requirements were satisfied without any changes to the core command or the management tool. The 20 lines of "hack" code did exactly what was needed and nothing more. Sometimes the most elegant solution is the one that acknowledges the constraints you're working within and finds a way to succeed despite them, rather than trying to change the constraints themselves.