When NFS Whispers: Uncovering a Multi-Layered APT in Plain Sight

The clock read 3:34 AM when the anomaly became impossible to ignore. Igor was poring over NFS traffic captures from a client's reverse gateway cluster, trying to correlate encrypted port 443 traffic with decrypted HTTP on port 80 to identify requests containing PII. Everything looked normal until he filtered for PII-associated connections—the packet timing distribution skewed sharply toward longer intervals, with no corresponding anomaly on the HTTP side. Then he noticed the EOF value in an NFS read reply was 77685 bytes, exactly 8192 bytes more than the requested read length. The extra data had uniform entropy, suggesting encryption, yet the file on disk wasn't encrypted. That's when the investigation shifted from routine assessment to incident response.

The gateways in question ran a minimalist stack: a monolithic kernel without modules and a statically linked Go application serving as both init replacement and reverse proxy. During boot, the Go app mounted a hardcoded NFS share containing configuration and TLS certificates. Igor's initial traffic analysis had revealed a useful detail—the gateway added an 'X-Orig-Connection' header with the TLS four-tuple, allowing him to map encrypted connections to their decrypted HTTP counterparts. But the NFS anomaly pointed to something deeper.

Comparing his test environment captures with the production mirror, Igor spotted a subtle protocol deviation: NFS open requests showed 'open-id:' instead of the expected 'open id:' string. Digging into the kernel source (NFS-ganesha 3.3), he found the hardcoded string in encode_openhdr() had been replaced by a call to unknown_func(), which sometimes substituted the space with a hyphen. While the NFS server ignored this opaque client identifier, the malware had weaponized it.

Memory forensics revealed the truth: a malicious libfsalvfs.so had been injected into the NFS-ganesha process, replacing the legitimate library. This backdoored FSAL implemented two covert channels:

1. Data exfiltration channel: When an NFS open request used 'open-id:' (note the hyphen), the file handle was marked. On subsequent reads, the malware appended a payload from its runtime storage to the final block and altered the on-wire EOF value to reflect the new size. Unpatched kernels ignored the extra bytes since they only checked EOF for zero/non-zero, but Igor's analysis caught the size discrepancy.

2. Command and control channel: Writes to paths like /<export>/.snapshot/meta/<cmdid> were intercepted by the malware instead of being passed to the underlying filesystem. Commands included:

   • 1704: Load a shared object from buffer (e.g., cp payload.so .snapshot/meta/1704)
   • 1702: Set auto-self-destruct time via UDP broadcast on port 41701
   • 74650-74652: Arm/disarm/payload management for the first covert channel

The second stage was even more clever. To inspect decrypted HTTP traffic without modifying the Go application directly, the attackers patched the kernel:

• Allocated RWX memory (trampoline area) after the Go binary
• Located net/http.(*connReader).Read via ELF symbol table
• Overwrote the function's preamble with a jump to the trampoline
• The trampoline examined the stack to find the return address, then allocated a second trampoline per call site that:
  • Copied the original mov reg, [rsp+?] instruction (which loads Read()'s result)
  • Replaced it with a jump to malware inspection code
  • Restored execution after inspection

This allowed the malware to see every decrypted HTTP response immediately after the Go stdlib's Read() returned, enabling PII scanning without touching the Go binary—bypassing any integrity checks on the application itself.

The persistence mechanism was equally stealthy. Kernel updates came from a CI/CD pipeline that built and signed kernels on an air-gapped segment, but the build process had temporarily shifted to a Windows laptop running VirtualBox. Attackers exploited this three-month window to compromise the build environment, signing their malicious kernel with the legitimate key. The Go application, built in a separate monitored segment with MFA, remained untouched—which explained why the hook lived in the kernel.

When Igor triggered the self-destruct command (1701) during forensics, it sent a UDP broadcast and restarted a clean NFS-ganesha instance—but not before the malware had already exfiltrated data via its covert channels. The client's insistence on restoring service quickly meant some volatile evidence was lost, but the reconstructed malware samples revealed impressive tradecraft: the string obfuscation function used a billion-nested switch tree that required brute-force reversal to extract indicators like /proc/self/mem paths and Ganesha log locations.

This case highlights how attackers can chain seemingly innocuous protocol quirks (like an ignored NFS identifier) into full compromise when combined with kernel-level persistence and application-layer inspection. For defenders, it underscores the need to monitor not just for known bad patterns, but for subtle protocol deviations and unexpected kernel modifications—even in "trusted" build pipelines. The full technical details, including YARA rules and memory dump analysis techniques, will follow in a subsequent post as Igor continues reversing the second-stage payload.