Abstract
Mobile edge computing requires more and more high-performance servers, resulting in increased energy consumption. As an effective means to reduce energy consumption, virtual machine placement (VMP) has been widely studied. In the edge computing environment, as the number of terminal device requests continues to increase, the scale of VMP becomes larger and larger, and existing research algorithms may take a long time to converge. The reason is that as the number of VMs increases, the search space of the policy becomes larger and the agent needs to interact with the environment for a longer time to make the best decision. In addition, existing research methods only consider reducing energy consumption, rarely consider the response latency of virtual machines, and almost ignore the dynamic changes of the edge environment. To overcome these drawbacks, we propose a virtual machine placement algorithm based on meta-reinforcement learning, which consists of an inner and outer loop. The inner loop designs a deep reinforcement learning algorithm combined with the order exchange and migration mechanism to generate the best decision, and the outer loop provides meta-strategy parameters for the inner loop based on meta-learning to accelerate the convergence capability of the inner loop, thereby obtaining efficient virtual machine placement decisions quickly from a new environment. Through simulation experiments, we demonstrate that our approach effectively reduces the energy consumption of the edge server and the response latency of VMs at different problem sizes compared to the three baseline algorithms. At the same time, it quickly adapts to the new environment with only a small number of gradient updates.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
The data used to support the findings of this study are available from the corresponding author upon request.
References
Dinh, H.T., Lee, C., Niyato, D., Wang, P.: A survey of mobile cloud computing: architecture, applications, and approaches. Wirel. Commun. Mob. Comput. 13, 1587–1611 (2013)
Taleb, T., Dutta, S., Ksentini, A., Iqbal, M., Flinck, H.: Mobile edge computing potential in making cities smarter. Comm. Mag. 55(3), 38–43 (2017). https://doi.org/10.1109/MCOM.2017.1600249CM
Ahmed, A., Ahmed, E.: A survey on mobile edge computing. In: International conference on intelligent systems & control (2016)
Satyanarayanan, M.: Mobile computing: the next decade. Mob. Comput. Commun. Rev. 15, 2–10 (2011)
Mosa, A., Sakellariou, R.: Dynamic virtual machine placement considering cpu and memory resource requirements. In: 2019 IEEE 12th international conference on cloud computing (CLOUD), pp. 196–198 (2019). https://doi.org/10.1109/CLOUD.2019.00042
Wen, C., Jiang, W.: Research on virtual machine layout strategy based on improved particle swarm optimization algorithm. In: 2019 IEEE 21st international conference on high performance computing and communications; IEEE 17th international conference on smart city; IEEE 5th international conference on data science and systems (HPCC/SmartCity/DSS), pp. 1343–1349 (2019). https://doi.org/10.1109/HPCC/SmartCity/DSS.2019.00187
Ding, Y., Liao, G., Liu, S.: Virtual machine placement based on degradation factor ant colony algorithm. In: 2018 13th IEEE conference on industrial electronics and applications (ICIEA), pp. 775–779 (2018). https://doi.org/10.1109/ICIEA.2018.8397818
Hospedales, T., Antoniou, A., Micaelli, P., Storkey, A.: Meta-learning in neural networks: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 44(9), 5149–5169 (2022). https://doi.org/10.1109/TPAMI.2021.3079209
Botvinick, M., Ritter, S., Wang, J., Kurth-Nelson, Z., Blundell, C., Hassabis, D.: Reinforcement learning, fast and slow. Trends Cogn. Sci. (2019). https://doi.org/10.1016/j.tics.2019.02.006
Jian, C., Bao, L., Zhang, M.: A high-efficiency learning model for virtual machine placement in mobile edge computing. Clust. Comput. 25(5), 3051–3066 (2022)
Li, S., Pan, X.: Adaptive management and multi-objective optimization of virtual machine in cloud computing based on particle swarm optimization. EURASIP J. Wirel. Commun. Netw. (2020). https://doi.org/10.1186/s13638-020-01722-4
Baalamurugan, K.M., Bhanu, S.V.: A multi-objective krill herd algorithm for virtual machine placement in cloud computing. J. Supercomput. 76(1), 4525–4542 (2020)
Jia, M., Cao, J., Liang, W.: Optimal cloudlet placement and user to cloudlet allocation in wireless metropolitan area networks. IEEE Trans. Cloud Comput. 5(4), 725–737 (2017). https://doi.org/10.1109/TCC.2015.2449834
Aghasi, A., Jamshidi, K., Bohlooli, A.: A thermal-aware energy-efficient virtual machine placement algorithm based on fuzzy controlled binary gravitational search algorithm (fc-bgsa). Clust. Comput. 25(2), 1015–1033 (2022)
Alharbe, N., Rakrouki, M.A., Aljohani, A.: An improved ant colony algorithm for solving a virtual machine placement problem in a cloud computing environment. IEEE Access 10, 44869–44880 (2022). https://doi.org/10.1109/ACCESS.2022.3170103
Li, Y., Wang, S.: An energy-aware edge server placement algorithm in mobile edge computing. In: 2018 IEEE International conference on edge computing (EDGE), pp. 66–73 (2018). https://doi.org/10.1109/EDGE.2018.00016
Caviglione, L., Gaggero, M., Paolucci, M., Ronco, R.: Deep reinforcement learning for multi-objective placement of virtual machines in cloud datacenters (vol 15, pg 613, 2020). Soft Comput. 19(25), 12569–12588 (2021)
Rizvi, N., Ramesh, D.: Fbq-la: fuzzy based q-learning approach for elastic workloads in cloud environment. J. Intell. Fuzzy Syst. 36, 1–14 (2018). https://doi.org/10.3233/JIFS-18828
Zeng, J., Ding, D., Kang, K., Xie, H., Yin, Q.: Adaptive drl-based virtual machine consolidation in energy-efficient cloud data center. IEEE Trans. Parallel Distrib. Syst. 33(11), 2991–3002 (2022). https://doi.org/10.1109/TPDS.2022.3147851
Ma, X., Xu, H., Gao, H., Bian, M., Hussain, W.: Real-time virtual machine scheduling in industry iot network: a reinforcement learning method. IEEE Trans. Ind. Inf. (2022). https://doi.org/10.1109/TII.2022.3211622
Wu, Y., Zhang, S., Shen, G., Chen, G.: Deep reinforcement learning for online vrc deployment in mobile edge computing. In: 2022 IEEE 23rd International conference on high performance switching and routing (HPSR), pp. 271–276 (2022). https://doi.org/10.1109/HPSR54439.2022.9831247
Sonmez, C., Ozgovde, A., Ersoy, C.: Edgecloudsim: An environment for performance evaluation of edge computing systems. In: 2017 Second international conference on fog and mobile edge computing (FMEC), pp. 39–44 (2017). https://doi.org/10.1109/FMEC.2017.7946405
Greenberg, A., Hamilton, J., Maltz, D.A., Patel, P.: The cost of a cloud: research problems in data center networks. ACM 39, 68–73 (2008)
Fan, X., Weber, W.D., Barroso, L.A.: Power provisioning for a warehouse-sized computer. Comput. Archit. News 35, 13–23 (2007)
Beloglazov, A., Abawajy, J., Buyya, R.: Energy-aware resource allocation heuristics for efficient management of data centers for cloud computing. Future Gener. Comput. Syst. 28(5), 755–768 (2012). https://doi.org/10.1016/j.future.2011.04.017. (Special section: energy efficiency in large-scale distributed systems)
Liu, X.-F., Zhan, Z.-H., Deng, J.D., Li, Y., Gu, T., Zhang, J.: An energy efficient ant colony system for virtual machine placement in cloud computing. IEEE Trans. Evolut. Comput. 22(1), 113–128 (2018). https://doi.org/10.1109/TEVC.2016.2623803
Huang, L., Bi, S., Zhang, Y.-J.A.: Deep reinforcement learning for online computation offloading in wireless powered mobile-edge computing networks. IEEE Trans. Mob. Comput. 19(11), 2581–2593 (2020). https://doi.org/10.1109/TMC.2019.2928811
Mnih, V., Kavukcuoglu, K., Silver, D., et al.: Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015). https://doi.org/10.1038/nature14236
Nichol, A., Achiam, J., Schulman, J.: On first-order meta-learning algorithms (2018). https://doi.org/10.48550/arXiv.1803.02999
Wang, J., Hu, J., Min, G., Zhan, W., Georgalas, N.: Computation offloading in multi-access edge computing using a deep sequential model based on reinforcement learning. IEEE Commun. Mag. 57(5), 64–69 (2019)
Funding
This work was supported in part by the National Natural Science Foundation of Chinaunder Grant No.61672461 and No.62073293.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors have not disclosed any competing interests.
Informed consent
Informed consent was obtained from all individual participants included in the study.
Research involved in human and animal participants
This article does not contain any studies with human participants or animals performed by any of the authors. Informed consent was obtained from all individual participants included in the study.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Xu, H., Jian, C. A meta reinforcement learning-based virtual machine placement algorithm in mobile edge computing. Cluster Comput 27, 1883–1896 (2024). https://doi.org/10.1007/s10586-023-04030-w
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10586-023-04030-w