Browse > Article
http://dx.doi.org/10.6109/jicce.2021.19.3.155

Cooperative Coevolution Differential Evolution Based on Spark for Large-Scale Optimization Problems  

Tan, Xujie (School of Computer and Big Data Science, Jiujiang University)
Lee, Hyun-Ae (School of Computer Information & Communication Engineering, Kunsan National University)
Shin, Seong-Yoon (School of Computer Information & Communication Engineering, Kunsan National University)
Publication Information
Journal of information and communication convergence engineering / v.19, no.3, 2021 , pp. 155-160 More about this Journal
Abstract
Differential evolution is an efficient algorithm for solving continuous optimization problems. However, its performance deteriorates rapidly, and the runtime increases exponentially when differential evolution is applied for solving large-scale optimization problems. Hence, a novel cooperative coevolution differential evolution based on Spark (known as SparkDECC) is proposed. The divide-and-conquer strategy is used in SparkDECC. First, the large-scale problem is decomposed into several low-dimensional subproblems using the random grouping strategy. Subsequently, each subproblem can be addressed in a parallel manner by exploiting the parallel computation capability of the resilient distributed datasets model in Spark. Finally, the optimal solution of the entire problem is obtained using the cooperation mechanism. The experimental results on 13 high-benchmark functions show that the new algorithm performs well in terms of speedup and scalability. The effectiveness and applicability of the proposed algorithm are verified.
Keywords
Cooperative coevolution; Differential evolution; Large-scale optimization; Resilient distributed datasets;
Citations & Related Records
연도 인용수 순위
  • Reference
1 D. Teijeiro, X. C. Pardo, P. Gonzalez, J. R. Banga, and R. Doallo, "Implementing parallel differential evolution on Spark," in Proceedings of European Conference on the Applications of Evolutionary Computation, Lecture Notes in Computer Science, vol. 9598, pp. 75-90, 2016. DOI:10.1007/978-3-319-31153-1_6.   DOI
2 R. A. Hasan, R. A. I. Alhayali, N. D. Zaki, and A. H. Ali, "An adaptive clustering and classification algorithm for Twitter data streaming in Apache Spark," Telkomnika, 2019, vol. 17, no. 6, pp. 3086-3099, 2019. DOI:10.12928/TELKOMNIKA.v17i6.11711.   DOI
3 X. Yao, Y. Liu, and G. Lin, "Evolutionary programming made faster," IEEE Transactions on Evolutionary Computation, vol. 3, no. 2, pp. 82-102, 1999. DOI:10.1109/4235.771163.   DOI
4 R. Storn and K. Price, "Differential evolution-a simple and efficient heuristic for global optimization over continuous spaces," Journal of Global Optimization, vol. 11, no. 4, pp. 341-359, 1997. DOI: 10.1023/A:1008202821328.   DOI
5 F. V. d. Bergh and A. P. Engelbrecht, "A cooperative approach to particle swarm optimization," IEEE Transactions on Evolutionary Computation, vol. 8, no. 3, pp. 225-239, 2004. DOI: 10.1109/TEVC.2004.826069.   DOI
6 Y. Mei, M. N. Omidvar, X. Li, and X. X. Yao, "A competitive divide-and-conquer algorithm for unconstrained large-scale blackbox optimization," ACM Transactions on Mathematical Software, vol. 42, no. 2, pp. 1-42, 2016. DOI: 10.1145/2791291.   DOI
7 X. Li and X. Yao, "Cooperatively coevolving particle swarms for large scale optimization," IEEE Transactions on Evolutionary Computation, vol. 16, no. 2, pp. 210-224, 2012. DOI: 10.1109/TEVC.2011.2112662.   DOI
8 L. Wan, G. Zhang, H. Li, and C. Li, "A novel bearing fault diagnosis method using Spark-based parallel ACO-K-means clustering algorithm," IEEE Access, vol. 9, pp. 28753-28768, 2021. DOI:10.1109/ACCESS.2021.3059221.   DOI
9 B. Liu, S. He, D. He, Y. Zhang, and M. Guizani, "A Spark-based parallel fuzzy c-means segmentation algorithm for agricultural image big data," in IEEE Access, vol. 7, pp. 42169-42180, 2019. DOI: 10.1109/ACCESS.2019.2907573.   DOI
10 C. Deng, X. Tan, X. Dong, and Y. Tan, "A parallel version of differential evolution based on resilient distributed datasets model," in Proceedings of Bio-Inspired Computing- Theories and Applications, Springer, Berlin, Heidelberg. vol. 562, pp 84-93, 2015. DOI: 10.1007/978-3-662-49014-3_8.   DOI
11 C. Zhou, "Fast parallelization of differential evolution algorithm using MapReduce," in Proceedings of the 12th annual conference on Genetic and evolutionary computation, pp. 1113-1114, 2010. DOI: 10.1145/1830483.1830689   DOI
12 K. Tagawa and T. Ishimizu, "Concurrent differential evolution based on MapReduce," International Journal of computers, vol. 4, no. 4, pp. 161-168, 2010.
13 H. Peng, X. Tan, C. Deng and S. Peng, "SparkCUDE: a spark-based differential evolution for large-scale global optimization," International Journal of High Performance Systems Architecture, vol. 7, no. 4, pp. 211-222, 2017. DOI: 10.1504/IJHPSA.2017.092390.   DOI
14 K. V. Price, R. M. Storn, and J. A. Lampinen, Differential evolution: A practical approach to global optimization, Springer, 2005.
15 X. W. Wang, Q. Y. Dai, W. C. Jiang, and J. Z. Cao, "Retrieval of design patent images based on MapReduce model," Journal of Chinese Computer Systems, vol. 33, no. 3, pp. 626-632, 2012.   DOI
16 Y. Wang, Z. Cai, and Q. Zhang, "Enhancing the search ability of differential evolution through orthogonal crossover," Information Sciences, vol. 185, no. 1, pp. 153-177, 2012. DOI: 10.1016/j.ins.2011.09.001.   DOI
17 J. Kennedy and R. Eberhart, "Particle swarm optimization," in Proceedings of ICNN'95 International Conference on Neural Networks, pp. 1942-1948, 1995. DOI: 10.1109/ICNN.1995.488968.   DOI
18 M. N. Omidvar, M. Yang, Y. Mei, X. Li, and X. Yao, "DG2: a faster and more accurate differential grouping for large-scale black-box optimization," IEEE Transactions on Evolutionary Computation, vol. 21, no. 6, pp. 929-942, 2017. DOI: 10.1109/TEVC.2017.2694221.   DOI
19 Y. Wang, Z. Cai, and Q. Zhang, "Differential evolution with composite trial vector generation strategies and control parameters," IEEE Transactions on Evolutionary Computation, vol 15, no. 1, pp. 55-66, 2011. DOI: 10.1109/TEVC.2010.2087271.   DOI
20 J. Brest, S. Greiner, B. Boskovic B, M. Mernik, and V. Zumer, "Self-adapting control parameters in differential evolution: a comparative study on numerical benchmark problems," IEEE Transactions on Evolutionary Computation, vol. 10, no. 6, pp. 646-657, 2006. DOI: 10.1109/TEVC.2006.872133.   DOI
21 M. A. Potter and K. A. De Jong, "A cooperative coevolutionary approach to function optimization," in Proceedings of International Conference on Parallel Problem Solving from Nature, vol. 866, pp. 249-257, 1994. DOI: 10.1007/3-540-58484-6_269.   DOI
22 X. Meng, J. Bradley, B. Yavuz, E. Sparks, S. Venkataraman, D. Liu, J. Freeman, DB Tsai, M. Amde, S. Owen, D. Xin, R. Xin, M. J. Franklin, R. Zadeh, M. Zaharia, and A. Talwalkar, "Mllib: machine learning in apache spark," The Journal of Machine Learning Research, vol. 17, no. 1, pp 1235-1241, 2016. DOI:10.5555/2946645.2946679.   DOI
23 Z. Yang, K. Tang, and X. Yao, "Large scale evolutionary optimization using cooperative coevolution," Information Sciences, vol. 178, no. 15, pp. 2985-2999, 2008. DOI: 10.1016/j.ins.2008.02.017.   DOI
24 X. Guan, X. Zhang, J. Wei, I. Hwang, Y. Zhu, and K. Cai, "A strategic conflict avoidance approach based on cooperative coevolutionary with the dynamic grouping strategy," International Journal of Systems Science, vol. 47, no. 9, pp. 1995-2008, 2016. DOI: 10.1080/00207721.2014.966282.   DOI
25 X. M. Hu, F. L. He, W. E. Chen, and J. Zhong, "Cooperation coevolution with fast interdependency identification for large scale optimization," Information Sciences, vol. 381, pp. 142-160, 2017. DOI: 10.1016/j.ins.2016.11.013.   DOI
26 Z. Jizhao, J. Yantao, X. Jun, Q. Jianzhong, W. Yuanzhuo,C. Xueqi, "SparkCRF: a parallel implementation of CRFs algorithm with spar," Journal of Computer Research and Development, vol. 53, no. 8. pp. 1819-1828, 2016. DOI: 10.7544/issn1000-1239.2016.20160197.   DOI
27 M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauley, M. J. Franklin, S. Shenker, and I. Stoica, "Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing," in Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation, pp. 15-28, 2012. DOI:10.5555/2228298.2228301.   DOI