Abstract
In the MapReduce processing, since map tasks output key-value pairs, and reduce tasks take the pairs output by the map tasks and compute the final results. Therefore, reduce tasks are unknown until their map tasks are finished. Also, we assume that map tasks are preemptive and parallelizable, but reduce tasks are non-parallelizable. With these assumptions, we study the scheduling of minimizing makespan. Both preemptive and non-preemptive reduce tasks are considered. We prove that no matter if preemption is allowed or not, any algorithm has a competitive ratio at least \(2-\frac{1}{h}\), we then give two optimal algorithms for these two versions.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Dean, J., Ghemawat, S.: Mapreduce: simplified data processing on large clusters. In: Proceedings of the 6th Conference on Symposium on Opearting Systems Design and Implementation, OSDI’04, vol. 6, p 10. USENIX Association, Berkeley (2004)
Isard, M., Prabhakaran, V., Currey, J., Wieder, U., Talwar, K., Goldberg, A.: Quincy: fair scheduling for distributed computing clusters. In: Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles, SOSP ’09, pp. 261–276. ACM, New York
Moseley, B., Dasgupta, A., Kumar, R., Sarlós, T.: On scheduling in map-reduce and flow-shops. In: Proceedings of the 23rd ACM Symposium on Parallelism in Algorithms and Architectures, SPAA ’11, pp. 289–298. ACM, New York (2011)
Sandholm, T., Lai, K.: Mapreduce optimization using regulated dynamic prioritization. In: Proceedings of the Eleventh International Joint Conference on Measurement and Modeling of Computer Systems, SIGMETRICS ’09, pp. 299–310. ACM, New York
White, T.: Hadoop: The Definitive Guide, 1st edn. O’Reilly Media Inc. (2009)
Zaharia, M., Borthakur, D., Sarma, J.S., Elmeleegy, K., Shenker, S., Stoica, I.: Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling. In: Proceedings of the 5th European Conference on Computer Systems, EuroSys ’10, pp. 265–278. ACM, New York
Zhu, Y., Jiang, Y., Wu, W., Ding, L., Teredesai, A., Li, D., Lee, W.: Minimizing makespan and total completion time in mapreduce-like systems. In: Proceedings of INFOCOM’14, IEEE (2014)
Acknowledgments
This work was supported in part under National Natural Science Foundation of China under Grants 61221063, 71371129 and Program for Changjiang Scholars and Innovative Research Team in University under Grant IRT1173.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Luo, T., Zhu, Y., Wu, W. et al. Online makespan minimization in MapReduce-like systems with complex reduce tasks. Optim Lett 11, 271–277 (2017). https://doi.org/10.1007/s11590-015-0902-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11590-015-0902-7