Bugtraq: The Malloc Maleficarum

The Malloc Maleficarum, a 2005 technical treatise posted to the Bugtraq mailing list, represents a watershed moment in the history of memory corruption exploitation. Authored by the pseudonymous "Phantasmal Phantasmagoria," this document cataloged a collection of sophisticated techniques for exploiting vulnerabilities in the GNU C Library's (glibc) malloc implementation. The work emerged as a direct response to a series of defensive patches applied to glibc in late 2004, which introduced over a dozen mandatory integrity checks designed to render existing exploitation methods obsolete. Rather than accepting defeat, the author presented a systematic framework for bypassing these new defenses, organized into six distinct "Houses" or methodologies.

The context for this work is critical. In the early 2000s, heap exploitation was dominated by techniques like "Vudo Malloc Tricks" (2001) and "Once Upon A free()" (2001), which primarily relied on corrupting the unlink() macro used during free operations. These methods were largely nullified by glibc 2.3.5's introduction of integrity assertions that validated chunk boundaries, sizes, and linkage pointers before performing operations. The Malloc Maleficarum documented how to circumvent these protections by targeting different, often overlooked, components of the malloc subsystem. Each House represents a distinct attack vector, requiring specific memory layouts and control over different aspects of the heap state.

The first technique, The House of Prime, focuses on corrupting the av->max_fast variable within the arena structure. The arena is a per-thread or global control structure that manages memory allocation. The technique requires two controlled free() calls followed by a malloc() call. The attacker first overflows a chunk to set its size to the minimum value of 8. When freed, this chunk is processed by the fastbin code path. Due to an integer underflow in the fastbin_index() macro ((8 >> 3) - 2 = -1), the forward pointer of the chunk is written to av->max_fast instead of a legitimate fastbin. A second free() call with a carefully chosen size then uses this corrupted av->max_fast to write a pointer to arena_key—a thread-specific variable that points to the current arena. By hijacking arena_key, subsequent malloc() calls are redirected to use a fake arena controlled by the attacker, enabling arbitrary memory reads or writes through further manipulation of the arena's bin structures.

The House of Mind provides a more direct path to arbitrary memory overwrite with a single free() call. It exploits the arena_for_chunk() macro, which determines which arena to use for a given chunk. If the chunk's size has the NON_MAIN_ARENA bit set, the macro uses heap_for_ptr() to locate a heap_info structure at a 1MB boundary. By carefully arranging the heap layout—typically through repeated allocations—the attacker can place a controlled buffer at such a boundary. This allows them to forge a fake heap_info structure containing an arbitrary ar_ptr (arena pointer). When free() is called on the overflowed chunk, this fake arena is passed to _int_free(), which then processes the chunk using the attacker's controlled arena structure. The attacker can then set up the arena's unsorted_chunks or fastbins to point to a target address (like a GOT entry), and the subsequent unlinking operations will overwrite that target with the address of the attacker's chunk.

The House of Force targets the "wilderness" or top chunk—the special chunk at the end of the heap that can be extended. This technique is unique in that it requires two malloc() calls and is useful when only the top chunk can be overflowed. The attacker first overflows the top chunk to set its size to the maximum possible value (e.g., 0xffffffff). Then, by calling malloc() with a specifically crafted, very large size, the wilderness code is tricked into setting the av->top pointer to an arbitrary location (the "remainder" chunk). This location is calculated as the current top chunk address plus the requested size. The attacker can use this to set av->top to a location just before a target like a GOT entry. A subsequent malloc() call for a large buffer will then return memory that overlaps the target, allowing the attacker to overwrite execution control data.

The House of Lore corrupts bin entries (small or large bins) rather than fastbins. It requires a chunk that has been freed and sorted into a bin, then overflowed. When a subsequent malloc() request matches the bin's size, the unlinking code removes the corrupted chunk from the bin. The attacker controls the bk (backward) pointer of the chunk, which is used to update the bin's bk pointer. A later malloc() request for the same size will then retrieve an arbitrary chunk from the corrupted bin. For small bins, the attacker must ensure the fake chunk's bk points to writable memory to avoid a crash during the unlinking process. For large bins, the technique is more complex due to the additional splitting logic, but the principle remains: corrupt the bin linkage to control the return value of future allocations.

The House of Spirit is notable for being applicable to both heap and stack overflows. It works by overwriting a pointer that will later be passed to free(). The attacker sets this pointer to a fake chunk header located near a target (e.g., a function pointer on the stack). The fake chunk's size must be crafted to pass integrity checks and place it in the correct fastbin. When free() is called on the corrupted pointer, the fake chunk is added to a fastbin. A subsequent malloc() for the same size will then return the fake chunk's address to the application. If the application then writes data to this chunk, it can overwrite the target. This technique requires precise control over the memory layout around the target but does not require corrupting any internal glibc structures directly.

Finally, The House of Chaos is less a technical exploit and more a philosophical manifesto. It frames the work in the context of "virtuality," where information is infinite and ownership is meaningless. The author, Phantasmal Phantasmagoria, positions themselves as a "virtual adept"—a consciousness existing purely in the virtual realm, creating and spreading information without regard for security, profit, or fame. This section serves as a declaration of intent, separating the technical work from conventional notions of hacking or security research. It emphasizes the pursuit of knowledge and the simplification of complex systems for the benefit of all conscious entities.

The Malloc Maleficarum's significance lies in its systematic approach to a rapidly evolving defensive landscape. It demonstrated that while individual mitigations could block specific techniques, the underlying complexity of malloc's implementation provided a fertile ground for new attack vectors. Each House required deep understanding of glibc's internal data structures—arenas, bins, chunks, and the wilderness—and highlighted the trade-offs between performance and security in memory allocators. The document remains a foundational reference for understanding heap exploitation, illustrating how attackers adapt to defenses by finding new paths through the same complex codebase. Its legacy endures in modern exploit development, where techniques like tcache poisoning (a successor to fastbin attacks) and house-of-style attacks continue to build upon the principles first articulated in this seminal work.