Jin Tack Lim
Jason Nieh
ABSTRACT
Nested virtualization, running virtual machines and hypervisors on top of other virtual machines and hypervisors, is increasingly important because of the need to deploy virtual machines running software stacks on top of virtualized cloud infrastructure. However, performance remains a key impediment to further adoption as application workloads can perform many times worse than native execution. To address this problem, we introduce DVH (Direct Virtual Hardware), a new approach that enables a host hypervisor, the hypervisor that runs directly on the hardware, to directly provide virtual hardware to nested virtual machines without the intervention of multiple levels of hypervisors. We introduce four DVH mechanisms, virtual-passthrough, virtual timers, virtual inter-processor interrupts, and virtual idle. DVH provides virtual hardware for these mechanisms that mimics the underlying hardware and in some cases adds new enhancements that leverage the flexibility of software without the need for matching physical hardware support. We have implemented DVH in the Linux KVM hypervisor. Our experimental results show that DVH can provide near native execution speeds and improve KVM performance by more than an order of magnitude on real application workloads.
- Nadav Amit, Muli Ben-Yehuda, Dan Tsafrir, and Assaf Schuster. 2017. vIOMMU: Efficient IOMMU Emulation. In Proceedings of the 2011 USENIX Annual Technical Conference (USENIX ATC 2011). Portland, OR, 105--121.Google Scholar
- ARM Ltd. 2011. ARM Generic Interrupt Controller Architecture version 2.0 ARM IHI 0048B.Google Scholar
- ARM Ltd. 2013. ARM Architecture Reference Manual ARMv8-A DDI0487A.a.Google Scholar
- ARM Ltd. 2016. ARM Generic Interrupt Controller Architecture Specification GIC architecture version 3.0 and version 4.0 ARM IHI 0069C.Google Scholar
- Jeff Barr. 2017. Now Available -- Compute-Intensive C5 Instances for Amazon EC2. AWS News Blog. Retrieved Jan 21, 2020 from https://aws.amazon.com/blogs/aws/now-available-computeintensive- c5-instances-for-amazon-ec2/Google Scholar
- Muli Ben-Yehuda, Michael D. Day, Zvi Dubitzky, Michael Factor, Nadav Har'El, Abel Gordon, Anthony Liguori, Orit Wasserman, and Ben-Ami Yassour. 2010. The Turtles Project: Design and Implementation of Nested Virtualization. In Proceedings of the 9th Symposium on Operating Systems Design and Implementation (OSDI 2010). Vancouver, BC, Canada, 423--436.Google Scholar
- Paolo Bonzini. 2018. Migration with directly assigned devices is possible? KVM Mailing List. Retrieved Jan 21, 2020 from https://marc.info/?l=kvm&m=152459004513285&w=2Google Scholar
- Edouard Bugnion, Jason Nieh, and Dan Tsafrir. 2017. Hardware and Software Support for Virtualization. Morgan and Claypool Publishers.Google Scholar
- Cesare Cantu. 2013. Network Interface Card Device Pass-through with Multiple Nested Hypervisors. US Patent US9176767B2.Google Scholar
- Sean Christopherson. 2019. KVM: nVMX: Disable intercept for FS/GS base MSRs in vmcs02 when possible. Linux Kernel Source Tree. Retrieved Jan 21, 2020 from https://github.com/torvalds/linux/ commit/d69129b4e46a7b61dc956af038d143eb791f22c7Google Scholar
- Citrix. 2020. Citrix and AWS partner to enable application elasticity and scale. Retrieved Jan 21, 2020 from https://www.citrix.com/global-partners/amazon-web-services/Google Scholar
- CloudShare. 2019. Infrastructure. Retrieved Jan 21, 2020 from https://www.cloudshare.com/technology/nested-virtualization/Google Scholar
- Christoffer Dall. 2018. The Design, Implementation, and Evaluation of the Linux ARM Hypervisor. Ph.D. Dissertation. Columbia University.Google Scholar
- Christoffer Dall, Shih-Wei Li, Jin Tack Lim, Jason Nieh, and Georgios Koloventzos. 2016. ARM Virtualization: Performance and Architectural Implications. In Proceedings of the 43rd International Symposium on Computer Architecture (ISCA 2016). Seoul, South Korea, 304--316.Google ScholarDigital Library
- Christoffer Dall, Shih-Wei Li, and Jason Nieh. 2017. Optimizing the Design and Implementation of the Linux ARM Hypervisor. In Proceedings of the 2017 USENIX Annual Technical Conference (USENIX ATC 2017). Santa Clara, CA, 221--234.Google ScholarDigital Library
- Christoffer Dall and Jason Nieh. 2013. KVM/ARM: Experiences Building the Linux ARM Hypervisor. Technical Report CUCS-010--13. Department of Computer Science, Columbia University.Google Scholar
- Christoffer Dall and Jason Nieh. 2014. KVM/ARM: The Design and Implementation of the Linux ARM Hypervisor. In Proceedings of the 19th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS 2014). Salt Lake City, UT, 333--347.Google ScholarDigital Library
- Yaozu Dong, Yu Chen, Zhenhao Pan, Jinquan Dai, and Yunhong Jiang. 2012. ReNIC: Architectural Extension to SR-IOV I/O Virtualization for Efficient Replication. ACM Transactions on Architecture and Code Optimization (TACO) 8, 4 (Jan. 2012), 40:1--40:22.Google Scholar
- DPDK. 2015. Poll Mode Driver for Emulated Virtio NIC. Retrieved Jan 21, 2020 from https://doc.dpdk.org/guides/nics/virtio.htmlGoogle Scholar
- DPDK. 2019. Data Plane Development Kit. Retrieved Jan 21, 2020 from https://dpdk.org/Google Scholar
- Joy Fan. 2017. Nested Virtualization in Azure. Azure Blog. Retrieved Jan 21, 2020 from https://azure.microsoft.com/en-us/blog/nestedvirtualization- in-azure/Google Scholar
- Bryan Ford, Mike Hibler, Jay Lepreau, Patrick Tullmann, Godmar Back, and Stephen Clawson. 1996. Microkernels Meet Recursive Virtual Machines. In Proceedings of the 2nd Symposium on Operating Systems Design and Implementation (OSDI 1996). Seattle,WA, 137--151.Google ScholarDigital Library
- Google Cloud. 2018. Enabling Nested Virtualization for VM Instances. Retrieved Jan 21, 2020 from https://cloud.google.com/compute/docs/ instances/enable-nested-virtualization-vm-instancesGoogle Scholar
- Abel Gordon, Nadav Amit, Nadav Har'El, Muli Ben-Yehuda, Alex Landau, Assaf Schuster, and Dan Tsafrir. 2012. ELI: Bare-metal Performance for I/O Virtualization. In Proceedings of the 17th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS 2012). London, England, UK, 411--422.Google ScholarDigital Library
- Stefan Hajnoczi. 2011. An Updated Overview of the QEMU Storage Stack. In LinuxCon Japan 2011. Yokohama, Japan.Google Scholar
- Intel Corporation. 2012. Intel 64 and IA-32 Architectures Software Developer's Manual, 325462-044US.Google Scholar
- Intel Corporation. 2013. 4th Generation Intel Core vPro Processors with Intel VMCS Shadowing. Retrieved Jan 21, 2020 from http://www.intel.com/content/dam/www/public/us/en/documents/ white-papers/intel-vmcs-shadowing-paper.pdfGoogle Scholar
- Rick Jones. 2010. Netperf. Retrieved Jan 21, 2020 from https://github.com/HewlettPackard/netperfGoogle Scholar
- Richard WM Jones. 2014. Super-nested KVM. Retrieved Jan 21, 2020 from https://rwmj.wordpress.com/2014/07/03/super-nested-kvm/Google Scholar
- Asim Kadav and Michael M. Swift. 2008. Live Migration of Directaccess Devices. In Proceedings of the 1st Workshop on I/O Virtualization (WIOV 2008). San Diego, CA.Google Scholar
- KVM. 2018. Tuning KVM - KVM,. Retrieved Jan 21, 2020 from https:// www.linux-kvm.org/index.php?title=Tuning_KVM&oldid=173911Google Scholar
- Alex Landau, Muli Ben-Yehuda, and Abel Gordon. 2011. SplitX: Split Guest/Hypervisor Execution on Multi-core. In Proceedings of the 3rd Workshop on I/O Virtualization (WIOV 2011). Portland, OR.Google Scholar
- Shih-Wei Li, John S. Koh, and Jason Nieh. 2019. Protecting Cloud Virtual Machines from Commodity Hypervisor and Host Operating System Exploits. In Proceedings of the 28th USENIX Security Symposium (USENIX Security 2019). Santa Clara, CA, 1357--1374.Google Scholar
- Wanpeng Li. 2018. KVM: X86: Provide a capability to disable HLT intercepts. Linux Kernel Source Tree. Retrieved Jan 21, 2020 from https://github.com/torvalds/linux/commit/ caa057a2cad647fb368a12c8e6c410ac4c28e063Google Scholar
- Cunming Liang and Tiwei Bie. 2018. vdpa: vhost-mdev as a New vhost Protocol Transport. In KVM Forum 2018. Edinburgh, Scotland, UK.Google Scholar
- Jin Tack Lim, Christoffer Dall, Shih-Wei Li, Jason Nieh, and Marc Zyngier. 2017. NEVE: Nested Virtualization Extensions for ARM. In Proceedings of the 26th ACM Symposium on Operating Systems Principles (SOSP 2017). Shanghai, China, 201--217.Google ScholarDigital Library
- Microsoft. 2009. Windows XP Mode. Retrieved Jan 21, 2020 from https://www.microsoft.com/en-us/download/details.aspx?id=8002Google Scholar
- Microsoft. 2017. Virtualization-based Security (VBS). Retrieved Jan 21, 2020 from https://docs.microsoft.com/en-us/windowshardware/ design/device-experiences/oem-vbsGoogle Scholar
- Timothy Prickett Morgan. 2016. Azure Stack Gives Microsoft Leverage Over AWS, Google. The Next Platform. Retrieved Jan 21, 2020 from https://www.nextplatform.com/2016/01/26/azure-stack-givesmicrosoft- leverage-over-aws-google/Google Scholar
- Oracle. 2019. Oracle Cloud Infrastructure Ravello Service. Retrieved Jan 21, 2020 from https://docs.oracle.com/en/cloud/iaas/ravellocloud/ index.htmlGoogle Scholar
- Steven Osman, Dinesh Subhraveti, Gong Su, and Jason Nieh. 2002. The Design and Implementation of Zap: A System for Migrating Computing Environments. In Proceedings of the 5th Symposium on Operating Systems Design and Implementation (OSDI 2002). Boston, MA, 361--376.Google ScholarDigital Library
- Zhenhao Pan, Yaozu Dong, Yu Chen, Lei Zhang, and Zhijiao Zhang. 2012. CompSC: Live Migration with Pass-through Devices. In Proceedings of the 8th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (VEE 2012). London, England, UK, 109--120.Google ScholarDigital Library
- Ravello Community. 2016. Nested virtualization: How to run nested KVM on AWS or Google Cloud. Ravello Blog. Retrieved Jan 21, 2020 from https://blogs.oracle.com/ravello/run-nested-kvm-on-awsgoogleGoogle Scholar
- Robert Ricci, Eric Eide, and The CloudLab Team. 2014. Introducing CloudLab: Scientific Infrastructure for Advancing Cloud Architectures and Applications. USENIX ;login: 39, 6 (Dec. 2014), 36--38.Google Scholar
- Rusty Russell. 2008. Virtio: Towards a De-facto Standard for Virtual I/O Devices. ACM SIGOPS Operating Systems Review 42, 5 (July 2008), 95--103.Google ScholarDigital Library
- Rusty Russell, Yanmin Zhang, Ingo Molnar, and David Sommerseth. 2008. Improve hackbench. Linux Kernel Mailing List. Retrieved Jan 21, 2020 from http://people.redhat.com/mingo/cfsscheduler/ tools/hackbench.cGoogle Scholar
- Simon Sharwood. 2017. AWS adopts home-brewed KVM as new hypervisor. The Register. Retrieved Jan 21, 2020 from https://www.theregister.co.uk/2017/11/07/aws_writes_new_kvm_ based_hypervisor_to_make_its_cloud_go_faster/Google Scholar
- Paul Sim. 2013. KVM Performance Optimization. Retrieved Jan 21, 2020 from https://www.slideshare.net/janghoonsim/kvmperformance- optimization-for-ubuntuGoogle Scholar
- SUSE. 2020. Disk Cache Modes. Retrieved Jan 21, 2020 from https://documentation.suse.com/sles/12-SP4/html/SLES-all/chacachemodes. htmlGoogle Scholar
- The Apache Software Foundation. 2015. ab - Apache HTTP server benchmarking tool. Retrieved Jan 21, 2020 from http://httpd.apache.org/docs/2.4/programs/ab.htmlGoogle Scholar
- Cheng-Chun Tu, Michael Ferdman, Chao-tung Lee, and Tzi-cker Chiueh. 2015. A Comprehensive Implementation and Evaluation of Direct Interrupt Delivery. In Proceedings of the 11th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (VEE 2015). Istanbul, Turkey, 1--15.Google ScholarDigital Library
- Lluís Vilanova, Nadav Amit, and Yoav Etsion. 2019. Using SMT to Accelerate Nested Virtualization. In Proceedings of the 46th International Symposium on Computer Architecture (ISCA 2019). Phoenix, AZ, 750--761.Google ScholarDigital Library
- Andrew Whitaker, Marianne Shaw, and Steven D. Gribble. 2002. "Scale and Performance in the Denali Isolation Kernel". In Proceedings of the 5th Symposium on Operating Systems Design and Implementation (OSDI 2002). Boston, MA, 195--209.Google ScholarCross Ref
- Dan Williams, Yaohui Hu, Umesh Deshpande, Piush K. Sinha, Nilton Bila, Kartik Gopalan, and Hani Jamjoom. 2016. Enabling Efficient Hypervisor-as-a-Service Clouds with Ephemeral Virtualization. In Proceedings of the 12th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (VEE 2016). Atlanta, GA, 79--92.Google ScholarDigital Library
- Xen Project wiki. 2014. Network Throughput and Performance Guide. Retrieved Jan 21, 2020 from http://wiki.xen.org/wiki/Network_ Throughput_and_Performance_GuideGoogle Scholar
- Xen Project wiki. 2018. Nested Virtualization in Xen. Retrieved Jan 21, 2020 from https://wiki.xenproject.org/wiki/Nested_Virtualization_ in_XenGoogle Scholar
- Xen Project wiki. 2019. Xen PCI Passthrough. Retrieved Jan 21, 2020 from https://wiki.xen.org/wiki/Xen_PCI_PassthroughGoogle Scholar
- Xin Xu and Bhavesh Davda. 2017. A Hypervisor Approach to Enable Live Migration with Passthrough SR-IOV Network Devices. ACM SIGOPS Operating Systems Review 51, 1 (Sept. 2017), 15--23.Google ScholarDigital Library
- Edwin Zhai, Gregory D. Cummings, and Yaozu Dong. 2008. Live Migration with Pass-through Device for Linux VM. In Proceedings of the 2008 Ottawa Linux Symposium (OLS 2008). Ottawa, ON, Canada, 261--267.Google Scholar
Index Terms
- Optimizing Nested Virtualization Performance Using Direct Virtual Hardware
Recommendations
NEVE: Nested Virtualization Extensions for ARM
SOSP '17: Proceedings of the 26th Symposium on Operating Systems PrinciplesNested virtualization, the ability to run a virtual machine inside another virtual machine, is increasingly important because of the need to deploy virtual machines running software stacks on top of virtualized cloud infrastructure. As ARM servers make ...
A Virtual CPU Scheduling Model for I/O Performance in Paravirtualized Environments
RACS '17: Proceedings of the International Conference on Research in Adaptive and Convergent SystemsParavirtualization manages virtual machines and virtual resources efficiently by the communication between the virtualization layer and modified guest OSes. In a paravirtual environment, the I/O response of a virtual machine is hard to approach that of ...
A Low Overhead and Reliable Nested Virtualization VMM for Cloud Computing
WISA '13: Proceedings of the 2013 10th Web Information System and Application ConferenceCommodity operating systems have already gained functionality of virtual machine monitor. Nested virtualization is needed to run these commodity operating systems as virtual machines. Furthermore, with nested virtualization technology, users can run a ...
Comments