AWS Enables Nested Virtualization on EC2 Instances

AWS has introduced support for running nested virtual machines within EC2 instances, a feature that has been highly requested by the developer community for years. The new capability allows customers to run KVM or Hyper-V hypervisors inside their virtualized EC2 instances, enabling use cases such as mobile app emulation, hardware simulation, and Windows Subsystem for Linux on Windows workstations.

Three-Layer Architecture

The nested virtualization feature creates a three-layer architecture. At the base level (L0), the physical AWS infrastructure runs the Nitro hypervisor. The EC2 instance itself acts as the middle layer (L1), running its own hypervisor. Finally, virtual machines can be launched inside that instance as the top layer (L2). This architecture exposes processor features such as Intel VT-x to the instance through the Nitro System, allowing it to run virtual machines internally.

Supported Instance Types and Hypervisors

The feature is available on C8i, M8i, and R8i instance types, which are the latest generation of compute, memory, and storage-optimized instances from AWS. Both KVM and Hyper-V are supported as the L1 hypervisors. However, Graviton instances are not currently supported, and the feature is not available on older instance families.

Community Demand and Industry Response

The announcement has been met with enthusiasm from the developer community, with many expressing relief that the long-awaited feature is finally available. Comments on social media and technical forums reflect the sentiment that this capability has been missing from AWS for years while competitors like Azure have supported it.

Michael Boulos, a software engineer at Google, noted that Google Cloud Platform had implemented nested virtualization years ago, while AWS customers had to find workarounds. The feature enables uniformity of operations and control, even if it means sacrificing some peak performance compared to bare-metal alternatives.

Use Cases and Applications

Developers can now use this feature for various applications that previously required bare-metal instances or external virtualization solutions. Mobile app developers can run emulators directly on EC2 instances, automotive developers can simulate in-car hardware, and Windows users can run WSL on Windows workstations without leaving the AWS environment.

Anurag Goel, founder and CEO of Render, highlighted another important use case: running Firecracker and other microVMs inside AWS VMs instead of expensive bare-metal instances. This opens up possibilities for isolation, Lambda-style runtimes, and comparing microVMs versus containers from a performance and density perspective.

Implementation and Limitations

Enabling nested virtualization is now an API option that can be set at launch time. For example, customers can specify the NestedVirtualization=enabled parameter when launching an instance. However, AWS still recommends that customers running workloads requiring hardware virtualization, sensitive to performance, or requiring low latency continue using bare-metal instances rather than relying on nested virtualization.

The feature is available in all AWS regions on supported instance types, making it accessible to a global customer base. The API-based implementation means that developers can programmatically enable this feature as part of their infrastructure-as-code workflows.

Historical Context and Workarounds

Before this announcement, the only option for running nested virtualization on EC2 was to use bare-metal instances, which are physical servers without a hypervisor. Many developers resorted to using EC2 Mac instances, the smallest and cheapest bare-metal option, as a workaround for their virtualization needs. This new feature eliminates the need for such workarounds and provides a more cost-effective solution for many use cases.

Performance Considerations

While nested virtualization provides flexibility and convenience, it's important to note that there may be performance overhead compared to running directly on bare metal. The additional layer of virtualization can introduce latency and reduce computational efficiency. However, for many development, testing, and simulation scenarios, this trade-off is acceptable in exchange for the operational benefits and cost savings.

The ability to run nested virtualization on standard EC2 instances rather than requiring specialized bare-metal hardware represents a significant step forward in making virtualization more accessible and cost-effective for AWS customers. As the technology matures and optimizations are made, the performance gap between nested and non-nested virtualization is likely to narrow further.