GNOME Fixes Screencasting Issue With H.264 Recordings Being ~18x Larger Than VP8

A critical bug in GNOME Shell's screen casting functionality has been resolved, addressing a significant inefficiency that caused H.264 recordings to balloon to approximately 18 times the file size of VP8 recordings without any corresponding quality improvement.

The VA-API H.264 Recording Problem

The issue stemmed from GNOME Shell's desktop screen casting and recording code failing to configure proper rate control for VA-API H.264 pipelines. This oversight resulted in massively inflated file sizes that offered no perceptible quality benefits to users.

According to reports, users recording their GNOME desktop using VA-API H.264 with Mesa drivers on Radeon graphics hardware were generating files around 12 Mbit/s in size. In stark contrast, the software-based VP8 fallback produced recordings of approximately 630 kbit/s while maintaining similar visual quality.

This discrepancy meant that H.264 recordings were consuming roughly 18 times more storage space than necessary, creating problems for users with limited disk space or those sharing recordings online.

Technical Root Cause

The fundamental problem was the absence of rate control configuration in the VA-API H.264 pipeline. Without proper rate limiting, the encoder was essentially given free rein to use as much bandwidth as needed, resulting in unnecessarily large files.

Rate control is a crucial aspect of video encoding that manages the trade-off between file size and quality. When properly configured, it ensures consistent quality while maintaining reasonable file sizes. The lack of this configuration in GNOME's VA-API implementation meant the encoder was producing files at maximum quality settings regardless of whether such fidelity was perceptible or necessary for screen recordings.

Impact and Timeline

Evidence suggests this bug has been present since GNOME 44, meaning users have been dealing with inefficient recordings for multiple major releases. The issue was particularly problematic for:

• Users with limited storage capacity
• Those sharing recordings online (due to larger file sizes)
• Systems with slower internet connections
• Anyone archiving screen recordings

The Fix

The resolution came in the form of a remarkably concise two-line code change that properly configures rate control for VA-API H.264 pipelines. This fix ensures that H.264 recordings now maintain reasonable file sizes while preserving the quality users expect from hardware-accelerated encoding.

Performance Implications

With this fix implemented, users can expect:

• Significantly reduced file sizes for H.264 recordings (approximately 18x smaller)
• No noticeable quality degradation compared to pre-fix recordings
• Better storage efficiency for archived screen recordings
• Faster upload times when sharing recordings online
• Reduced bandwidth usage during recording sessions

Broader Context

This fix highlights the importance of proper configuration in hardware-accelerated video encoding. While VA-API provides excellent performance benefits through hardware acceleration, these advantages can be negated by improper software configuration.

The resolution also demonstrates GNOME's commitment to addressing performance issues, even those that may have persisted across multiple releases. The relatively simple nature of the fix underscores how small configuration oversights can have outsized impacts on user experience.

Moving Forward

For users who have been avoiding H.264 recording due to the file size issues, this fix should make hardware-accelerated recording a viable option again. The combination of VA-API's performance benefits with properly configured rate control should provide the best of both worlds: efficient file sizes with the performance advantages of hardware encoding.

This fix will be included in upcoming GNOME releases, bringing relief to users who rely on screen recording for tutorials, demonstrations, or documentation purposes.