The Hidden Complexity of PMU Counters on Apple Silicon

The pursuit of granular performance monitoring on Apple Silicon processors uncovers a landscape where hardware capabilities clash with undocumented constraints. Performance Monitoring Unit (PMU) counters – hardware registers tracking microarchitectural events like cache misses and branch predictions – promise deep insight into application behavior. Yet as detailed in Bugsik's investigation, their practical implementation reveals layers of complexity that challenge conventional profiling approaches.

At the core lies Apple's private kperf framework, reverse-engineered through work like ibireme's scoop library. This undocumented interface exposes 60+ counters on M-series chips, but imposes non-obvious constraints:

1. Fixed Limits: Only 10 counters can be monitored simultaneously, with two reserved for fixed-cycle and instruction events.
2. Group Conflicts: Six counters (dubbed Group M: INST_ALL, INST_INT_ALU, etc.) conflict pairwise due to identical hardware masks (0b0010000000).
3. Combinatorial Explosion: Adding counters beyond pairs reveals 18 Group G counters (BRANCH_COND_MISPRED_NONSPEC, INST_BARRIER, etc.) that conflict in sets larger than three. Attempts to catalog incompatibilities for 7-counter sets yielded 18 million failure cases.
4. Order Sensitivity: Counter addition order determines success, as hardware slot allocation depends on bitmask sequencing. For example, adding INST_LDST before ST_UNIT_UOP fails despite both being valid individually.

This behavior stems from the underlying slot-allocation algorithm: Each counter's 10-bit mask defines compatible hardware registers. When added sequentially, the system scans from right to left for available slots. Wide masks (like Group G's 0b1111111100) consume multiple slots, potentially blocking subsequent counters with overlapping requirements. Fixed counters (FIXED_CYCLES and FIXED_INSTRUCTIONS) avoid conflicts by using unique bit patterns (0b0000000001 and 0b0000000010).

The implications extend beyond academic curiosity:

• Tooling Limitations: Apple's Instruments app silently fails when exceeding invisible constraints, misleading developers.
• Optimization Blindspots: Inability to simultaneously monitor key metrics (e.g., cache misses + branch predictions) obscures performance bottlenecks.
• Research Burden: Absent official documentation, developers must reverse-engineer behavior through trial and error.

Bugsik's resulting tool, Lauka, pragmatically navigates these constraints by:

1. Prioritizing counter addition order
2. Exposing all available events via kpep_db structures
3. Providing benchmark comparisons with PMU metrics

This investigation highlights a broader tension in modern systems: Hardware complexity increasingly requires software to adapt to opaque physical constraints. While Apple's optimization guide offers high-level advice, the absence of low-level PMU documentation forces developers into combinatorial labyrinths. Alternative approaches like Arm's PMU specification demonstrate how transparency reduces such friction.

Ultimately, Bugsik's journey underscores that performance analysis on proprietary platforms demands equal parts tenacity and humility. The discovered constraints aren't arbitrary – they reflect tangible hardware resource limitations – but their obscurity transforms routine profiling into a research expedition. As silicon complexity grows, such hidden taxonomies may increasingly define the boundary between observable and inscrutable system behavior.