On the Performance of Cloud-based Virtual Machines

Cloud computing and virtualization solutions allow us to rent the virtual machines (VMs) needed to run applications on a pay-per-use basis, but rented VMs do not offer any guarantee on their performance: a slow VM may imply a bad response time at application level. One cannot assume a homogeneous and always predictable behavior of used VMs and understanding how VMs behaves over time and what guarantees they can offer is thus crucial to foresee the guarantees VM-based software can offer and to rent and allocate resources properly. This paper proposes a new empirical study based on VMs hosted in Europe and offered by the popular cloud infrastructures (Amazon  AWS, Microsoft Azure, and Google Cloud), and by EGI, a well-known European research-oriented federation of cloud providers. The assessment uses general-purpose VMs and 10 benchmarks to collect data on 28 different metrics. The paper also proposes PQI (Performance Quality Indicator), an indicator to get a synthetic measure of VM performance. As expected, and already studied by others, the study demonstrates that VM performance varies quite significantly, but to the best of our knowledge, this is the widest analysis ever conducted and provides interesting insights on how variability materializes and changes over time.