When Undefined Behavior Meets Retro Emulation: Inside 86Box’s Cross-Libc fseek Crash

Retro PC emulation is supposed to recreate 1990s chaos, not your production toolchain.

Yet that’s exactly what happened to 86Box—a hardware-accurate IBM PC emulator—when a single unchecked file pointer collided with differing interpretations of the C standard across glibc, BSD libc, Microsoft’s Universal CRT, and musl. On some systems, everything "just worked." On others, the emulator detonated on startup.

This isn’t only a bug tale. It’s a sharp reminder to systems developers: if you lean on undefined behavior, your real dependency isn’t a C standard library—it’s luck.

Source: This article is based on the investigation described by PVS-Studio in their post: “fseek in 86Box: Exploring the consequences of undefined behavior in different C libraries”.

The Setup: A Quiet Time Bomb in 86Box

86Box aims for hardware-accurate emulation of classic PCs: chipsets, buses, graphics boards, all modeled closely enough to run historical software as it was.

To support this, it persists video adapter NVRAM (non-volatile memory) into binary files. On startup, 86Box attempts to:

1. Open the NVRAM file.
2. If it doesn’t exist, create it with default data.
3. Seek and write as needed.

In one ATI 8514/A-related code path, the logic went wrong. The code attempted to use fseek on a file pointer (fp) that was never successfully opened. The pointer was NULL—and the return value of fseek was discarded via a cast to void.

In other words:

• fp could be NULL.
• fseek(fp, ...) was called unconditionally.
• The result was ignored.

Nothing exotic. No intricate race. Just a textbook null pointer misuse.

And in C, what happens next depends entirely on where you run it.

The Standard That Didn’t Save You

According to C11, the behavior of fseek with a NULL FILE* is not defined. The standard does not mandate a safety check, a diagnostic, or any particular failure mode.

That vacuum of responsibility gets filled by implementation detail.

• If your libc is defensive, you might get graceful failure.
• If it’s lean and assumes valid input, you get instant undefined behavior.
• If it’s somewhere in between, it might vary by build options or optimization levels.

The 86Box bug emerged precisely because the code implicitly assumed a benevolent implementation—but shipped to environments that made no such promise.

Four C Libraries, Four Behaviors

To understand the blast radius, the PVS-Studio team replayed the same scenario across major C library implementations.

1. glibc on MinGW: Silent Tolerance

On a Windows build using MinGW and glibc-variant behavior, fseek(NULL, ...) did not take the process down in their tests.

Through macros like CHECK_FILE, the implementation effectively guarded against invalid input in this configuration. The NVRAM file was created, the emulator booted, and nothing appeared wrong.

This permissiveness masked the underlying flaw.

2. glibc on GNU/Linux: Configuration-Sensitive Behavior

On a Devuan GNU/Linux system, glibc behaved less forgivingly—and crucially, behavior varied based on how glibc itself was built.

A mailing list discussion uncovered that certain configurations removed or altered internal checks. In those cases, passing a NULL FILE* could lead to undefined behavior, including process termination.

Same API. Same code. Different glibc build. Different fate.

The lesson: “It worked on my machine” is not a test plan; it’s an implementation detail.

3. BSD libc: Immediate Meltdown

On FreeBSD, things escalated.

The failing call chain looked roughly like this:

• fseek(fp, ...)
• → FLOCKFILE_CANCELSAFE(fp)
• → _FLOCKFILE(fp)
• → _flockfile(fp)

With fp == NULL, BSD libc dereferenced the null pointer while trying to lock the file structure. Result: abnormal exit and a core dump.

This was not a theoretical edge case; this was a deterministic crash when running 86Box under FreeBSD with the missing NVRAM file scenario.

4. Microsoft Universal CRT (UCRT): Strict and Loud

Microsoft’s Universal CRT takes a different approach: it validates parameters aggressively.

Internally, macros like _UCRT_VALIDATE_RETURN reject invalid file handles and treat them as fatal in release builds. Passing NULL to fseek leads to:

An invalid parameter was passed to a function that considers invalid parameters fatal.

In other words: no undefined shrug, no quiet fallback—just a deliberate abort.

5. musl: Lean and Unsafe-by-Design

musl, built for simplicity and correctness over defensive handholding, followed a pattern similar to BSD libc in this context:

• fseek → __fseeko → FLOCK macro
• The FILE* is dereferenced without a prior NULL check.

With a null pointer, behavior is predictably catastrophic—and fully consistent with the implementation’s expectations of well-formed input.

The Fix: One Line, Many Lessons

The 86Box bug wasn’t subtle once identified by PVS-Studio’s diagnostic:

V575 The null pointer is passed into 'fseek' function. Inspect the first argument. vid_ati_eeprom.c 61

The offending sequence attempted to seek on fp before it was (re)opened, in the code path handling a missing NVRAM file.

The correct behavior was already present nearby: another function handling a similar EEPROM case reopened the file cleanly before use.

The practical fix:

• Remove the fseek call on a potentially null fp.
• Let the code reopen and initialize the file explicitly before any file operation.

Once patched and rebuilt, FreeBSD—previously the most explosive testbed—ran cleanly. No crashes, no dumps, just a BIOS setup screen on a virtual PS/2 55SX with an emulated ATI 8514/A.

The issue was resolved in the 86Box 5.1 release, making the emulator behave reliably "out of the box" on problematic platforms.

Why This Matters to Every Systems Developer

This story is more than a quirky emulator bug.

1. Undefined Behavior Is an ABI Contract You Didn’t Mean to Sign

When you ignore a function’s preconditions, you are—implicitly—binding your software to a particular libc’s quirks, build flags, and internal macros.

The 86Box team didn’t intend to target “MinGW-glibc-with-this-exact-configuration-only,” but the combination of:

• unchecked FILE* usage,
• (void)fseek(...) silencing error handling,
• reliance on one environment’s leniency,

meant that’s effectively what they shipped.

For portable C and C++ code, assuming undefined behavior will "probably" fail gracefully is architectural debt.

2. Cross-Platform Means Cross-Libc

For developers shipping to:

• Linux distros mixing glibc and musl,
• BSDs in various roles (appliances, storage, networking),
• Windows with UCRT and different compiler stacks,

your real target matrix is not just "OS x Compiler"—it’s OS x Compiler x C Library x Build Configuration.

If you:

• discard return values from low-level APIs,
• skip precondition checks,
• rely on behavior outside the standard,

then you’re testing only a fraction of what your users will actually run.

3. Static Analysis Isn’t Optional Anymore

PVS-Studio’s role here is not incidental. A human reviewer might skim past (void)fseek(fp, 0, SEEK_END); and miss that fp can be NULL in a specific branch.

Static analyzers:

• do not get bored,
• do not normalize suspicious patterns as "probably fine," and
• can model paths across platforms and build modes.

For projects like emulators, hypervisors, databases, network daemons—anything close to the metal—systematic static analysis is now table stakes.

From Retro Bugs to Modern Discipline

The charm of 86Box is fidelity: registers, timings, interrupts, the quirks of physical boards long out of production. Ironically, that same ethos needs to apply to its host-side correctness.

This incident underlines a set of practices that every serious C/C++ team should internalize:

• Treat every undefined behavior as a potential cross-platform fault injector.
• Never trust unspecified behavior to "fail safe" across libraries.
• Wire static analysis into CI and treat its high-confidence findings as non-negotiable.
• Remember that libc implementations are not interchangeable runtime skins; they’re independent, opinionated systems.

86Box’s null pointer wasn’t just a defect; it was an x-ray of how fragile our assumptions become when they leave the boundaries of the standard and wander into "it worked once on my laptop."

In an ecosystem where your code might run atop glibc today, musl tomorrow, and UCRT in a compatibility layer next year, discipline is no longer optional. It’s infrastructure.