A deep dive into how OpenSSH's keystroke timing obfuscation feature generates hundreds of packets per keystroke, creating unexpected overhead for high-performance applications running over SSH, and the technical path to mitigating it.
The discovery began with an anomaly that seemed impossible. While debugging a high-performance game server running over SSH, a broken test harness that sent no game data still consumed 50% of the expected CPU. This paradox led to an investigation into the fundamental mechanics of SSH itself, revealing a feature designed for privacy that creates significant overhead for latency-sensitive applications.
The Packet Explosion
When analyzing network traffic for a single keystroke sent over SSH, the pattern is startling. A typical session shows hundreds of packets flowing for what should be a minimal data transfer. In one captured trace, a single keystroke generated 270 packets, with 179 of them being exactly 36 bytes each. The data payload itself was minimal—just 6,444 bytes in those 36-byte messages, plus 564 bytes of other data. This represents an 11.4x overhead ratio where the protocol chatter vastly outweighs the actual data being transmitted.
The timing pattern is equally revealing. These packets arrive approximately every 11.1 milliseconds, creating a steady stream of network activity that persists even when no actual user data is being transmitted. For applications that need to maintain thousands of concurrent connections, this overhead compounds rapidly.
The Root Cause: Keystroke Timing Obfuscation
The mystery unravels when examining OpenSSH's debug output. The verbose logging reveals a feature called "obfuscate_keystroke_timing" that activates with each keystroke, sending "chaff" packets at approximately 20-millisecond intervals. The debug messages show patterns like "chaff time expired (101 chaff packets sent)" for a single keystroke event.
This feature was introduced in OpenSSH 9.3 in 2023. The design rationale is straightforward: the timing between keystrokes can reveal information about what a user is typing. Different letters require different amounts of thought, and the rhythm of typing can betray patterns. By flooding the connection with chaff packets that mimic keystroke timing, SSH makes it difficult for an observer to distinguish real keystrokes from noise.
For traditional SSH sessions—where privacy is paramount and latency is less critical—this trade-off makes sense. The overhead is acceptable when the alternative is exposing sensitive information about typing patterns. However, for real-time applications like games, collaborative tools, or high-frequency data streams, this overhead becomes a significant performance bottleneck.
The Performance Impact
In the context of a game server targeting 2,000 concurrent players updating a 80x60 display 10 times per second, the math becomes stark. Each connection generates hundreds of packets per second just from keystroke obfuscation, even when players aren't actively typing. For 2,000 players, this translates to hundreds of thousands of packets per second of pure overhead.
The CPU impact is substantial. In the documented case, disabling keystroke obfuscation reduced total CPU usage from 29.90% to 11.64%—a 61% reduction. System call overhead dropped from 3.10 seconds to 0.66 seconds, and cryptographic operations decreased from 1.6 seconds to 0.11 seconds. Network bandwidth usage also improved, dropping from approximately 6.5 Mbit/sec to 3 Mbit/sec.
The Technical Solution
The chaff packets are actually SSH transport protocol messages: SSH2_MSG_PING and SSH2_MSG_PONG. These messages use numbers in the "local extensions" number space and are advertised using a "[email protected]" extension message. The server advertises support for this extension, and the client sends PING messages that the server must acknowledge with PONG responses.
The solution, therefore, is elegantly simple: don't advertise the [email protected] extension. The go-crypto library's SSH implementation includes this feature, and the code to add it is minimal—just a few lines that advertise the extension during the handshake.
Reverting this change in the go-crypto library and updating dependencies to use the modified version eliminates the chaff packets entirely. The application still functions normally because the ping extension is optional. SSH connections establish and data flows without the overhead of timing obfuscation.
Implementation Considerations
Forking a core cryptographic library like go-crypto introduces maintenance challenges. The fork must be kept up-to-date with security patches and upstream changes. However, the modification is surgical—only the extension advertisement needs to be removed. The actual cryptographic operations, key exchange, and data transport remain untouched.
For applications that need to support both traditional SSH sessions and real-time applications, a more sophisticated approach might be necessary. This could involve:
- Application-level detection: Identifying when a connection is being used for real-time applications versus traditional terminal sessions
- Configuration options: Providing a way for applications to request obfuscation be disabled
- Protocol negotiation: Extending SSH to allow clients and servers to negotiate whether obfuscation should be used
Broader Implications
This discovery highlights a fundamental tension in protocol design: security features that protect one use case can hinder another. SSH was designed primarily for secure remote terminal access, where privacy and security are paramount. The keystroke obfuscation feature serves this purpose well.
However, SSH has evolved into a general-purpose secure transport protocol. It's used for file transfers (SFTP), port forwarding, tunneling, and as a foundation for applications like the game described here. These applications have different requirements—lower latency, higher throughput, and predictable performance.
The challenge is that SSH's design philosophy prioritizes security and privacy over performance. This is generally correct, but it creates friction when the protocol is used in contexts where those priorities differ.
The LLM-Assisted Debugging Experience
The investigation also provides insight into using large language models for technical debugging. The author used Claude Code to analyze packet captures and explore hypotheses, finding it valuable for quickly parsing complex data and suggesting directions for investigation.
However, the process wasn't entirely automated. The initial insight—that the overhead persisted even with a broken test harness—required human intuition. Similarly, the final leap to forking the cryptographic library required understanding the trade-offs and maintenance implications that an LLM might not fully appreciate.
The experience suggests that LLMs excel at pattern recognition and hypothesis generation but still require human judgment for architectural decisions and understanding broader context. The debugging process becomes a collaboration where the human provides domain knowledge and strategic direction, while the LLM accelerates data analysis and exploration.
Conclusion
SSH's keystroke timing obfuscation represents a well-intentioned security feature that creates significant overhead for real-time applications. The solution—disabling the ping extension advertisement—is technically straightforward but introduces maintenance considerations.
For developers building high-performance applications over SSH, this discovery is crucial. The overhead isn't a bug or inefficiency in their code; it's a deliberate feature of the protocol. Understanding this allows for informed decisions about whether to accept the overhead, modify the protocol implementation, or consider alternative transports.
The broader lesson is that protocol features designed for one use case can create unexpected challenges when the protocol is repurposed. As SSH continues to evolve from a terminal protocol to a general-purpose secure transport, these tensions will likely emerge more frequently, requiring careful consideration of trade-offs and potentially protocol extensions to support diverse requirements.
For the game developer in this story, the discovery led to a >50% performance improvement—transformative gains that came not from optimizing their application code, but from understanding and modifying the underlying transport protocol's behavior.
Related Resources:
- OpenSSH Release Notes - Details on keystroke timing obfuscation
- go-crypto SSH Implementation - Source code for the SSH implementation
- SSH Protocol Extensions - Technical specification of SSH transport protocol
- Bubbletea TUI Framework - The TUI library mentioned in the article
- Wish SSH Server Library - SSH server library for Go applications
Comments
Please log in or register to join the discussion