Achieving a 0-CVE OS for VMs: The End of Traditional Patching

The promise of zero known vulnerabilities (0-CVE) in production systems has long been the holy grail of infrastructure security. While containerized workloads have made significant strides in this direction through immutable, versioned deployments, virtual machines have remained stubbornly tied to traditional patching cycles—a process that is inherently risky, difficult to reverse, and often leaves systems in an inconsistent state. The core problem isn't just the vulnerabilities themselves, but the operational model that makes addressing them so fraught with uncertainty.

Traditional VM patching follows a familiar, anxiety-inducing pattern. A system administrator runs dnf update or apt upgrade, hoping the application will survive the change. If it doesn't, the rollback process is manual, error-prone, and often requires snapshots that may not capture the full state. This fear creates a culture of delay, where security patches are postponed, increasing the window of exposure. The fundamental issue is that the operating system and its applications are treated as mutable, stateful entities that drift over time, making every update a unique event with unpredictable outcomes.

The solution, as I discovered through years of wrestling with this problem, lies in treating the operating system itself as an immutable artifact—exactly like a container image. RHEL's image mode, also known as bootable containers, provides this capability by allowing you to build a complete bootable disk image using standard Dockerfile syntax. The workflow is remarkably similar to container development: you define your base image, harden it, add applications, and build a versioned, immutable artifact. This artifact can then be deployed to any environment—AWS AMI, VMware VMDK, or raw disk image—through the bootc-image-builder tool.

The power of this approach becomes evident when examining the operational model. Instead of patching a running system, you build a new, hardened image and deploy it as a complete unit. The update process is atomic: you pull the new image, switch to it, and reboot. If anything goes wrong, rolling back is as simple as booting into the previous version, which remains intact and available. This eliminates the "snowflake server" problem where each machine becomes unique through manual configuration changes, and it provides the confidence needed for organizations to actually apply security updates promptly.

In my implementation, I've structured the approach around a repository model that mirrors container development practices. Each environment (rhel9, rhel10, python-3.14, jre) has its own Dockerfile and baselayout directory containing configuration files, systemd units, and scripts. The baselayout approach is critical—it separates the immutable OS from the mutable state. Only specific paths like /etc and /var remain writable for configuration and persistent data, while the core system remains read-only. This separation is what enables reliable rollbacks and prevents configuration drift.

To maintain the 0-CVE promise, I've implemented an automated pipeline using a custom tool called digestbot. Similar to Renovatebot or Dependabot, it monitors official Red Hat base images for updates. When a new version is detected, it automatically triggers rebuilds of all hardened base images and dependent application images, then runs comprehensive tests including compliance checks and vulnerability scans. This daily automation ensures that security patches are applied systematically rather than manually, removing human delay from the equation.

The end-user experience is intentionally simple. Application owners receive a script that checks for available updates and provides the exact bootc update and bootc switch commands needed. The cognitive load is minimal—no more worrying about dependency conflicts or manual rollback procedures. For teams ready for full automation, I'm developing a system where AWS instance tags like example.com/prefer-maintenance-window and example.com/soft-apply: yes can trigger automatic updates during specified windows, with health checks validating application functionality before committing the change.

This approach fundamentally changes the security posture of VM workloads. By treating the OS as an immutable, versioned artifact, we achieve the same security guarantees as containers while maintaining the flexibility of virtual machines. The attack surface is minimized through deliberate package selection, updates are atomic and reversible, and compliance becomes a matter of image validation rather than runtime inspection. The operational confidence this provides—knowing that any update can be instantly reverted—breaks the adoption barrier that has kept organizations locked in risky, manual patching cycles for decades.

For organizations considering this transition, the key insight is that the technology now exists to bridge the container and VM worlds. RHEL image mode provides the tooling, but the real transformation is philosophical: moving from patching mutable systems to deploying immutable, versioned artifacts. The implications extend beyond security into operational reliability, developer experience, and compliance automation. While not every workload may be suitable for this model, for the majority of server applications, it represents a fundamental improvement in how we manage and secure our infrastructure.