The 2008 arrest of Terry Childs, the San Francisco network administrator who held the city's FiberWAN hostage, became an IT cautionary tale of rogue admins and digital extortion. But an exclusive account from a direct colleague, speaking anonymously to Infoworld, reveals a far more nuanced tragedy—one where institutional failure and technological complexity collided with a fiercely dedicated Cisco Certified Internetwork Expert (CCIE).

The Architect and His Masterpiece

Childs wasn't merely an admin; he was the sole architect and builder of San Francisco's mission-critical FiberWAN—an MPLS network renowned for its complexity. His colleague's account paints a picture of profound technical mastery meeting dangerous isolation:

"Terry was the only person who fully understood the FiberWAN configuration... He locked everyone else out. I know most of the networking equipment does use centralized AAA, but I get the impression he may have configured the FiberWAN equipment for local authentication only."

This wasn't subterfuge, but an open secret tolerated by management for years. Departments knew requests impacting the FiberWAN "had to wait for Terry." His peers acknowledged the network's fragility under less skilled hands, even whispering the dreaded "bus factor" question: "What if Terry gets hit by a truck?"

Dedication Turned Toxic

Childs embodied the archetype of the overburdened, underappreciated senior engineer:
* 24/7 Ownership: As the city's only CCIE, he worked nights, weekends, and remained on perpetual standby without overtime pay.
* Architectural Control: He bypassed standard MPLS conventions, tightly controlling both Provider Edge (PE) and Customer Edge (CE) routers to maintain dominance over routing tables.
* Managerial Alienation: Childs clashed fiercely with non-technical managers he viewed as obsessed with bureaucracy over functionality, dismissing demands for documentation as obstructive to real work.

The Breaking Point

Tensions exploded when new security leadership demanded access to the FiberWAN. Childs, convinced others would "screw it up and bring the City network crashing to a halt," allegedly fortified his position—deleting backups and hardening access. His arrest followed swiftly. Crucially, the network remained operational post-arrest, supporting Childs' claim of protective intent, not sabotage.

Institutional Negligence Laid Bare

The insider reserves harshest criticism for city leadership:
* No Security Policy: "It’s only been a month or so since the City even had an information security policy... a bare, unmodified template." Childs had reportedly submitted policies for years, rejected because management "didn’t want to be held to it."
* No Contingency Planning: Despite years of sole administrator reliance, cross-training and backup access protocols were never enforced.
* Operational Risk Ignored: Managers accepted the single point of failure despite Childs' known temperament and burnout signs.

The Enduring Lesson

Childs' actions were extreme, but the conditions enabling them persist: complex systems entrusted to irreplaceable individuals without oversight or redundancy. The $5 million bail reflected fear, not damage—the network ran flawlessly without him. The real cost was operational paralysis: an inability to modify critical infrastructure and the loss of their top engineer.

"It’s a real shame. The city is losing a good network engineer — probably the best, technically, that they’ve ever had. Ultimately he has no one to blame but himself, but it’s too bad his superiors weren’t better about establishing and enforcing policies."

This case remains a stark monument to the dangers of valuing technical prowess over governance. It asks every IT leader: Where is your FiberWAN? Who truly holds the keys? And what happens when brilliance becomes a bottleneck?

Source: Why San Francisco's network admin went rogue (InfoWorld)