KSII Transactions on Internet and Information Systems (TIIS)

Korean Society for Internet Information (한국인터넷정보학회)
권호 표지
DOI QR코드

DOI QR Code

A Systematic Literature Review of Test Case Prioritization Technique on Software Product Line Testing

  • Siti Hawa Mohamed Shareef (Faculty of Computer Science and Information Technology, Universiti Tun Hussein Onn Malaysia) ;
  • Rabatul Aduni Sulaiman (Faculty of Computer Science and Information Technology, Universiti Tun Hussein Onn Malaysia) ;
  • Abd Samad Hasan Basari (Faculty of Computer Science and Information Technology, Universiti Tun Hussein Onn Malaysia)
  • Received : 2023.07.17
  • Accepted : 2024.10.08
  • Published : 2024.10.31
Download PDF
⟨ Previous Next ⟩

Abstract

Software product lines (SPL) leverage extensive reuse to enhance quality and competitiveness. However, the inherent high testing risks underscore the need for efficient techniques. Test case prioritization (TCP) emerges as a pivotal strategy for improving defect detection by optimizing the arrangement of test cases, thereby maximizing benefits, particularly when testing is prematurely halted. This paper presents a comprehensive review of the latest TCP techniques, investigating emerging trends, varied approaches, and the associated cost-effectiveness trade-offs. Through a systematic literature review spanning from 2011 to 2023 and employing specific search terms, this study scrutinizes the existing body of research. The analysis reveals a growing demand for TCP in research over the past five years, accompanied by a positive upsurge in SPL testing trends. TCP proves instrumental in orchestrating test cases from high to low priority levels, facilitating early defect detection and subsequent error resolution. Nonetheless, notable gaps persist in terms of time execution and coverage, prompting ongoing research efforts aimed at enhancing the cost-effectiveness of TCP. While TCP serves as a potent regression technique in software testing, existing approaches stand to benefit from the reordering of test cases and the integration of refined methodologies gleaned from current research endeavors.

Keywords

Acknowledgement

This research was supported by Ministry of Higher Education (MOHE) through Fundamental Research Grant Scheme (FRGS/1/2022/ICT01/UTHM/03/2).

References

  1. R. Rabiser, P. O'Leary, and I. Richardson, "Key activities for product derivation in software product lines," Journal of Systems and Software, vol.84, no.2, pp.285-300, Feb. 2011. https://doi.org/10.1016/j.jss.2010.09.042
  2. J. A. Prado Lima, W. D. F. Mendonca, S. R. Vergilio, and W. K. G. Assuncao, "Cost-effective learning-based strategies for test case prioritization in continuous integration of highly-configurable software," Empir Softw Eng., vol.27, no.6, Jul. 2022.
  3. M. Idham, S. A. Halim, D. N. A. Jawawi, Z. Zakaria, A. Erianda, and N. Arss, "Test Case Prioritization for Software Product Line: A Systematic Mapping Study," JOIV: International Journal on Informatics Visualization, vol.7, no.3-2, pp.2126-2134, 2023. https://doi.org/10.30630/joiv.7.3-2.1340
  4. P. Jung, S. Kang, and J. Lee, "Automated code-based test selection for software product line regression testing," Journal of Systems and Software, vol.158, Dec. 2019.
  5. M. Khatibsyarbini, M. A. Isa. D. N.A. Jawawi, and R. Tumeng, "Test case prioritization approaches in regression testing: A systematic literature review," Information and Software Technology, vol.93, pp.74-93, 2018.
  6. S. Yoo and M. Harman, "Regression testing minimization, selection and prioritization: a survey," Software Testing, Verification and Reliability, vol.22, no.2, pp.67-120, 2012. https://doi.org/10.1002/stvr.430
  7. O. Dahiya and K. Solanki, "A systematic literature study of regression test case prioritization approaches," International Journal of Engineering and Technology, vol.7, no.4, pp.2184-2191, 2018.
  8. A. Bajaj and O. P. Sangwan, "A Systematic Literature Review of Test Case Prioritization Using Genetic Algorithms," IEEE Access, vol.7, pp.126355-126375, 2019. https://doi.org/10.1109/ACCESS.2019.2938260
  9. A. Dadwal, H. Washizaki, Y. Fukazawa, T. Iida, M. Mizoguchi, and K. Yoshimura, "Prioritization in Automotive Software Testing: Systematic Literature Review," in Proc. of 6th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2018), 2018.
  10. P. Achimugu, A. Selamat, R. Ibrahim, and M. N. Mahrin, "A systematic literature review of software requirements prioritization research," Inf Softw Technol., vol.56, no.6, pp.568-585, 2014. https://doi.org/10.1016/j.infsof.2014.02.001
  11. J. Guo, J. White, G. Wang, J. Li, and Y. Wang, "A genetic algorithm for optimized feature selection with resource constraints in software product lines," Journal of Systems and Software, vol.84, no.12, pp.2208-2221, Dec. 2011. https://doi.org/10.1016/j.jss.2011.06.026
  12. Z. Anwar, H. Afzal, N. Bibi, H. Abbas, A. Mohsin, and O. Arif, "A hybrid-adaptive neuro-fuzzy inference system for multi-objective regression test suites optimization," Neural Comput & Applic., vol.31, no.11, pp.7287-7301, Nov. 2019. https://doi.org/10.1007/s00521-018-3560-8
  13. W. K. G. Assuncao, T. E. Colanzi, S. R. Vergilio, and A. Pozo, "A multi-objective optimization approach for the integration and test order problem," Information Sciences, vol.267, pp.119-139, May 2014. https://doi.org/10.1016/j.ins.2013.12.040
  14. R. A. Matnei Filho and S. R. Vergilio, "A multi-objective test data generation approach for mutation testing of feature models," Journal of Software Engineering Research and Development, vol.4, no.1, Dec. 2016.
  15. S. Zamani and H. Hemmati, "A pragmatic approach for hyper-parameter tuning in search-based test case generation," Empir Software Eng., vol.26, no.6, Nov. 2021.
  16. P. Oliveira, M. Souza, R. Braga, R. Britto, R. L. Rabelo, and P. S. Neto, "Athena: A Visual Tool to Support the Development of Computational Intelligence Systems," in Proc. of 2014 IEEE 26th International Conference on Tools with Artificial Intelligence ICTAI, pp.950-959, 2014.
  17. J. A. Galindo, D. Benavides, P. Trinidad, A.-M. Gutierrez-Fernandez, and A. Ruiz-Cortes, "Automated analysis of feature models: Quo vadis?," Computing, vol.101, no.5, pp.387-433, May 2019. https://doi.org/10.1007/s00607-018-0646-1
  18. B. J. Garvin, M. B. Cohen, and M. B. Dwyer, "Evaluating improvements to a meta-heuristic search for constrained interaction testing," Empir Software Eng., vol.16, no.1, pp.61-102, Feb. 2011. https://doi.org/10.1007/s10664-010-9135-7
  19. Hitesh and A. C. Kumari, "Feature Selection Optimization in SPL using Genetic Algorithm," Procedia Comput Sci., vol.132, pp.1477-1486, 2018. https://doi.org/10.1016/j.procs.2018.05.082
  20. Y. Xue et al., "IBED: Combining IBEA and DE for optimal feature selection in software product line engineering," Applied Soft Computing, vol.49, pp.1215-1231, Dec. 2016. https://doi.org/10.1016/j.asoc.2016.07.040
  21. H. Almulla and G. Gay, "Learning how to search: generating effective test cases through adaptive fitness function selection," Empir Software Eng., vol.27, no.2, Mar. 2022.
  22. S. Wang, D. Buchmann, S. Ali, A. Gotlieb, D. Pradhan, and M. Liaaen, "Multi-objective test prioritization in software product line testing: an industrial case study," in Proc. of SPLC '14: Proceedings of the 18th International Software Product Line Conference, pp.32-41, Association for Computing Machinery, Sep. 2014.
  23. M. W. Mkaouer, M. Kessentini, S. Bechikh, M. O Cinneide, and K. Deb, "On the use of many quality attributes for software refactoring: a many-objective search-based software engineering approach," Empir Software Eng, vol.21, no.6, pp.2503-2545, Dec. 2016. https://doi.org/10.1007/s10664-015-9414-4
  24. H. L. Jakubovski Filho, T. N. Ferreira, and S. R. Vergilio, "Preference based multi-objective algorithms applied to the variability testing of software product lines," Journal of Systems and Software, vol.151, pp.194-209, May 2019. https://doi.org/10.1016/j.jss.2019.02.028
  25. B. Perez Lamancha, M. Polo, and M. Piattini, "PROW: A Pairwise algorithm with constRaints, Order and Weight," Journal of Systems and Software, vol.99, pp.1-19, Jan. 2015. https://doi.org/10.1016/j.jss.2014.08.005
  26. T. Ferreira, S. R. Vergilio, and M. Kessentini, "Variability testing of software product line: A preference-based dimensionality reduction approach," Inf Softw Technol., vol.152, Dec. 2022.
  27. C. Birchler, S. Khatiri, P. Derakhshanfar, S. Panichella, A. Panichella, "Single and Multi-objective Test Cases Prioritization for Self-driving Cars in Virtual Environments," ACM Transactions on Software Engineering and Methodology, vol.32, no.2, pp.1-30, Apr. 2023.
  28. K. Pohl and A. Metzger, "Software Product Lines," The Essence of Software Engineering, pp.185-201, 2018.
  29. T. K. Akila and A. Malathi, "Test case prioritization using modified genetic algorithm and ant colony optimization for regression testing," International Journal of Advanced Technology and Engineering Exploration, vol.9, no.88, pp.384-400, Mar. 2022.
  30. R. A. Sulaiman, D. N. A. Jawawi, and S. A. Halim, "A Dissimilarity with Dice-Jaro-Winkler Test Case Prioritization Approach for Model-Based Testing in Software Product Line," KSII Transactions on Internet and Information Systems, vol.15, no.3, pp.932-951, Mar. 2021.
  31. F. Damiani, D. Faitelson, C. Gladisch, and S. Tyszberowicz, "A novel model-based testing approach for software product lines," Softw Syst Model., vol.16, no.4, pp.1223-1251, Oct. 2017. https://doi.org/10.1007/s10270-016-0516-2
  32. H. Lackner and B.-H. Schlingloff, "Advances in Testing Software Product Lines," Advances in Computers, vol.107, pp.157-217, 2017. https://doi.org/10.1016/bs.adcom.2017.07.001
  33. S. Segura, A. B. Sanchez, and A. Ruiz-Cortes, "Automated variability analysis and testing of an E-commerce site.: an experience report," in Proc. of ASE '14: Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering, pp.139-150, Association for Computing Machinery, Inc, 2014.
  34. H. Andersson, E. Herzog, and J. Olvander, "Experience from model and software reuse in aircraft simulator product line engineering," Inf Softw Technol., vol.55, no.3, pp.595-606, Mar. 2013. https://doi.org/10.1016/j.infsof.2012.06.014
  35. C. D. N. Damasceno, M. R. Mousavi, and A. da S. Simao, "Learning by sampling: learning behavioral family models from software product lines," Empir Software Eng., vol.26, no.1, Jan. 2021.
  36. L. Mariani, M. Pezze, and D. Zuddas, "Recent Advances in Automatic Black-Box Testing," Advances in Computers, vol.99, pp.157-193, 2015. https://doi.org/10.1016/bs.adcom.2015.04.002
  37. P. Runeson and E. Engstrom, "Regression Testing in Software Product Line Engineering," Advances in Computers, vol.86, pp.223-263, 2012. https://doi.org/10.1016/B978-0-12-396535-6.00007-7
  38. S. Lity, M. Nieke, T. Thum, and I. Schaefer, "Retest test selection for product-line regression testing of variants and versions of variants," Journal of Systems and Software, vol.147, pp.46-63, Jan. 2019. https://doi.org/10.1016/j.jss.2018.09.090
  39. R. Lindohf, J. Kruger, E. Herzog, and T. Berger, "Software product-line evaluation in the large," Empir Software Eng., vol.26, no.2, Mar. 2021.
  40. M. Lochau, J. Burdek, S. Holzle, and A. Schurr, "Specification and automated validation of staged reconfiguration processes for dynamic software product lines," Softw Syst Model., vol.16, no.1, pp.125-152, Feb. 2017. https://doi.org/10.1007/s10270-015-0470-4
  41. X. Devroey et al., "Statistical prioritization for software product line testing: an experience report," Softw Syst Model., vol.16, no.1, pp.153-171, Feb. 2017. https://doi.org/10.1007/s10270-015-0479-8
  42. R. Lachmann, S. Lity, M. Al-Hajjaji, F. Furchtegott, and I. Schaefer, "Fine-grained test case prioritization for integration testing of delta-oriented software product lines," in Proc. of FOSD 2016: Proceedings of the 7th International Workshop on Feature-Oriented Software Development, pp.1-10, Association for Computing Machinery, Inc, Oct. 2016.
  43. R. A. B. Sulaiman, "Cost-effective model-based test case generation and prioritization for software product line," PhD thesis, Universiti Teknologi Malaysia, Faculty of Engineering - School of Computing, 2020.
  44. F. Di Silvestro, "Improving testing reusability and automation for software product lines," M.S. thesis, Dept. Innovation Design and Eng., Vasteras, Sweden, 2020.
  45. H. Samih, H. Le Guen, R. Bogusch, M. Acher, and B. Baudry, "An Approach to Derive Usage Models Variants for Model-Based Testing," in Proc. of 26th IFIP WG 6.1 International Conference, ICTSS 2014, Testing Software and Systems, LNPSE, vol.8763, pp.80-96, 2014.
  46. M. Felderer, W. Hasselbring, H. Koziolek, F. Matthes, L. Prechelt, R. Reussner, B. Rumpe, and I. Schaefer Eds, Ernst Denert Award for Software Engineering 2020, Springer Cham, 2020.
  47. S. A. Halim, D. N. A. Jawawi, and M. Sahak, "Similarity Distance Measure and Prioritization Algorithm for Test Case Prioritization in Software Product Line Testing," Journal of Information and Communication Technology, vol.18, no.1, pp.57-75, 2019. https://doi.org/10.32890/jict2019.18.1.4
  48. T. Lavoie, M. Merineau, E. Merlo, and P. Potvin, "A case study of TTCN-3 test scripts clone analysis in an industrial telecommunication setting," Inf Softw Technol., vol.87, pp.32-45, Jul. 2017. https://doi.org/10.1016/j.infsof.2017.01.008
  49. S. A. Razak, M. A. Isa, and D. N. A. Jawawi, "A Comparison on Similarity Distances and Prioritization Techniques for Early Fault Detection Rate," Journal of Telecommunication, Electronic and Computer Engineering, vol.9, no.3-3, pp.89-94, 2017.
  50. R. Huang et al., "Abstract Test Case Prioritization Using Repeated Small-Strength Level-Combination Coverage," IEEE Transactions on Reliability, vol.69, no.1, pp.349-372, Mar. 2020. https://doi.org/10.1109/TR.2019.2908068
  51. M. Sahak, D. N. A. Jawawi, and S. A. Halim, "An Experiment of Different Similarity Measures on Test Case Prioritization for Software Product Lines," Journal of Telecommunication, Electronic and Computer Engineering, vol.9, no.3-4, pp.177-185, 2017.
  52. M. Al-Hajjaji, T. Thum, M. Lochau, J. Meinicke, and G. Saake, "Effective product-line testing using similarity-based product prioritization," Softw Syst Model., vol.18, no.1, pp.499-521, Feb. 2019. https://doi.org/10.1007/s10270-016-0569-2
  53. N. Gokilavani and B. Bharathi, "Multi-Objective based test case selection and prioritization for distributed cloud environment," Microprocessors and Microsystems, vol.82, Apr. 2021.
  54. C. D. N. Damasceno, P. C. Masiero, and A. Simao, "Similarity testing for role-based access control systems," Journal of Software Engineering Research and Development, vol.6, no.1, Jan. 2018.
  55. D. Flemstrom, P. Potena, D. Sundmark, W. Afzal, and M. Bohlin, "Similarity-based prioritization of test case automation," Software Quality Journal, vol.26, no.4, pp.1421-1449, Dec. 2018. https://doi.org/10.1007/s11219-017-9401-7
  56. S. Khoshmanesh and R. Lutz, "Does Link Prediction Help Detect Feature Interactions in Software Product Lines (SPLs)?," arXiv:2009.07392, 2020.
  57. S. Segura, R. M. Hierons, D. Benavides, and A. Ruiz-Cortes, "Automated metamorphic testing on the analyses of feature models," Inf Softw Technol., vol.53, no.3, pp.245-258, Mar. 2011. https://doi.org/10.1016/j.infsof.2010.11.002
  58. M. Aggarwal and S. Sabharwal, "Combinatorial Test Set Prioritization Using Data Flow Techniques," Arab J Sci Eng., vol.43, no.2, pp.483-497, Feb. 2018. https://doi.org/10.1007/s13369-017-2631-y
  59. A. B. Sanchez and S. Segura, "SmarTest: A Test Case Prioritization Tool for Drupal," in Proc. of SPLC '17: Proceedings of the 21st International Systems and Software Product Line Conference, vol.B, pp.9-12, Association for Computing Machinery, Sep. 2017.
  60. A. B. Sanchez, S. Segura, J. A. Parejo, and A. Ruiz-Cortes, "Variability testing in the wild: the Drupal case study," Softw Syst Model., vol.16, no.1, pp.173-194, Feb. 2017. https://doi.org/10.1007/s10270-015-0459-z
  61. M. Sahak, S. Abd Halim, D. N. A. Jawawi, and M. A. Isa, "Evaluation of Software Product Line Test Case Prioritization Technique," International Journal on Advanced Science Engineering Information Technology, vol.7, no.4-2, pp.1601-1608, 2017. https://doi.org/10.18517/ijaseit.7.4-2.3403
  62. A. B. Sanchez, S. Segura, and A. Ruiz-Cortes, "A Comparison of Test Case Prioritization Criteria for Software Product Lines," in Proc. of 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation, pp.41-50, 2014.
  63. H. Hemmati, "Advances in Techniques for Test Prioritization," Advances in Computers, vol.112, pp.185-221, 2019. https://doi.org/10.1016/bs.adcom.2017.12.004
  64. A. Arrieta, G. Sagardui, L. Etxeberria, and J. Zander, "Automatic generation of test system instances for configurable cyber-physical systems," Software Quality Journal, vol.25, no.3, pp.1041-1083, Sep. 2017. https://doi.org/10.1007/s11219-016-9341-7
  65. U. Markiegi, A. Arrieta, L. Etxeberria, and G. Sagardui, "Dynamic test prioritization of product lines: An application on configurable simulation models," Software Quality Journal, vol.29, no.4, pp.943-988, Dec. 2021. https://doi.org/10.1007/s11219-021-09571-0
  66. C. Birchler, S. Khatiri, B. Bosshard, A. Gambi, and S. Panichella, "Machine learning-based test selection for simulation-based testing of self-driving cars software," Empirical Software Engineering, vol.28, Apr. 2023.
  67. I. Reinhartz-Berger and M. Kemelman, "Extracting core requirements for software product lines," Requirements Engineering, vol.25, no.1, pp.47-65, Mar. 2020. https://doi.org/10.1007/s00766-018-0307-0
  68. R. Lachmann, S. Beddig, S. Lity, S. Schulze, and I. Schaefer, "Risk-based integration testing of software product lines," in Proc. of VaMoS '17: Proceedings of the 11th International Workshop on Variability Modelling of Software-Intensive Systems, pp.52-59, 2017.
  69. H. Jahan, Z. Feng, and S. M. H. Mahmud, "Risk-Based Test Case Prioritization by Correlating System Methods and Their Associated Risks," Arabian Journal for Science and Engineering, vol.45, no.8, pp.6125-6138, Aug. 2020. https://doi.org/10.1007/s13369-020-04472-z
  70. R. Huang, Q. Zhang, T. Y. Chen, J. Hamlyn-Harris, D. Towey, and J. Chen, "An Empirical Comparison of Fixed-Strength and Mixed-Strength for Interaction Coverage Based Prioritization," IEEE Access, vol.6, pp.68350-68372, 2018. https://doi.org/10.1109/ACCESS.2018.2879638
  71. R. Huang et al., "On the Selection of Strength for Fixed-Strength Interaction Coverage Based Prioritization," in Proc. of 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC), pp.310-315, 2018.
  72. H. Akimoto, Y. Isogami, T. Kitamura, N. Noda, and T. Kishi, "A Prioritization Method for SPL Pairwise Testing Based on User Profiles," in Proc. of 2019 26th Asia-Pacific Software Engineering Conference (APSEC), pp.118-125, Dec. 2019.
  73. A. Ensan, E. Bagheri, M. Asadi, D. Gasevic, and Y. Biletskiy, "Goal-Oriented Test Case Selection and Prioritization for Product Line Feature Models," in Proc. of 2011 8th International Conference on Information Technology: New Generations, ITNG 2011, pp.291- 298, 2011.
  74. S. Kumar, Rajkumar, and M. Rani, "Collaborative Filtering-based Test Case Prioritization and Reduction for Software Product-Line Testing," in Proc. of IEEE Region 10 International Conference (TENCON), pp.498-503, Oct. 2019.
  75. A. K. Thurimella and B. Brugge, "A mixed-method approach for the empirical evaluation of the issue-based variability modeling," Journal of Systems and Software, vol.86, no.7, pp.1831-1849, 2013. https://doi.org/10.1016/j.jss.2013.01.038
  76. A. Arrieta, S. Segura, U. Markiegi, G. Sagardui, and L. Etxeberria, "Spectrum-based fault localization in software product lines," Inf Softw Technol., vol.100, pp. 18-31, Aug. 2018. https://doi.org/10.1016/j.infsof.2018.03.008
  77. F. C. Ferrari, A. Rashid, and J. C. Maldonado, "Towards the practical mutation testing of AspectJ programs," Science of Computer Programming, vol.78, no.9, pp.1639-1662, Sep. 2013. https://doi.org/10.1016/j.scico.2013.02.011
  78. I. Hajri, A. Goknil, F. Pastore, and L. C. Briand, "Automating system test case classification and prioritization for use case-driven testing in product lines," Empirical Software Engineering, vol.25, no.5, pp.3711-3769, Sep. 2020. https://doi.org/10.1007/s10664-020-09853-4
  79. L. Teixeira, P. Borba, and R. Gheyi, "Safe composition of configuration knowledge-based software product lines," Journal of Systems and Software, vol.86, no.4, pp.1038-1053, Apr. 2013. https://doi.org/10.1016/j.jss.2012.11.006
  80. Z. Chen et al., "Exploring Better Black-Box Test Case Prioritization via Log Analysis," ACM Transactions on Software Engineering and Methodology, vol.32, no.3, pp.1-32, Apr. 2023.
  81. T. Landsberg, C. Dietrich, and D. Lohmann, "TASTING: Reuse Test-case Execution by Global AST Hashing," in Proc. of the 17th International Conference on Software Technologies (ICSOFT 2022), vol.1, pp.33-45, 2022.
  82. S. Ali et al., "Towards Pattern-Based Change Verification Framework for Cloud-Enabled Healthcare Component-Based," IEEE Access, vol.8, pp.148007-148020, 2020. https://doi.org/10.1109/ACCESS.2020.3014671
  83. H.-F. Guo and Z. Qiu, "Automatic Grammar-Based Test Generation," in Proc. of 25th IFIP WG 6.1 International Conference, ICTSS 2013, Testing Software and Systems, LNPSE, vol.8254, pp.17-32, 2013.
  84. G. Goos et al., Fundamental Approaches to Software Engineering, 2022.