Browse > Article
http://dx.doi.org/10.3837/tiis.2022.05.008

Classification Trends Taxonomy of Model-based Testing for Software Product Line: A Systematic Literature Review  

Sulaiman, Rabatul Aduni (Department of Software Engineering, Faculty of Computer Science and Information System, Universiti Tun Hussein Onn Malaysia)
Jawawi, Dayang Norhayati Abang (Department of Software Engineering, School of Computing, Faculty Engineering Universiti Teknologi Malaysia)
Halim, Shahliza Abdul (Department of Software Engineering, School of Computing, Faculty Engineering Universiti Teknologi Malaysia)
Publication Information
KSII Transactions on Internet and Information Systems (TIIS) / v.16, no.5, 2022 , pp. 1561-1583 More about this Journal
Abstract
Context: Testing is one of the techniques that can assure the quality of software including the domain of Software Product Line (SPL). Various techniques have been deliberated to enhance the quality of SPL including Model-based Testing (MBT). Objective: The objective of this study is to analyze and classify trends of MBT in SPL covering the solutions, issues and evaluation aspects by using taxonomy form. Method: A Systematic Literature Review (SLR) was conducted involving 63 primary studies from different sources. The selected studies were categorized based on their common characteristics. Results: Several findings can guide future research on MBT for SPL. The important finding is that the multiple measurements are still open to improving current metrics to evaluate test cases in MBT for SPL. The multiple types of measurement required a trade-off between maximization and minimization results to ensure the testing method which could satisfy multiple test criteria for example cost and effectiveness at the same time.
Keywords
model-based; model-based testing; software product line; variability;
Citations & Related Records
Times Cited By KSCI : 1  (Citation Analysis)
연도 인용수 순위
1 X. Devroey, "Behavioural Model Based Testing of Software Product Lines," in Proc. of Softw. Prod. Line Conf., no. August, pp. 1-8, 2014.
2 U. Kelter, C. Pietsch, T. Kehrer, D. Reuling, and M. Ohrndorf, "SiPL -- A Delta-Based Modeling Framework for Software Product Line Engineering," in Proc. of 30th IEEE/ACM Int. Conf. Autom. Softw. Eng., pp. 852-857, 2015.
3 M. Lochau, S. Lity, R. Lachmann, I. Schaefer, and U. Goltz, "Delta-oriented model-based integration testing of large-scale systems," J. Syst. Softw., vol. 91, no. 1, pp. 63-84, 2014.   DOI
4 J. Lee, S. Kang, and P. Jung, "Test coverage criteria for software product line testing: Systematic literature review," Inf. Softw. Technol., vol. 122, no. January, p. 106272, 2020.   DOI
5 M. Azanza and L. Montalvillo, "20 years of Industrial Experience at SPLC : a Systematic Mapping Study," in Proc. of Int. Syst. Softw. Prod. Line Conf., vol. A, pp. 172-183, 2021.
6 K. L. Petry, E. OliveiraJr, and A. F. Zorzo, "Model-based testing of software product lines: Mapping study and research roadmap," J. Syst. Softw., vol. 167, 2020.
7 I. D. C. Machado, J. D. McGregor, Y. C. Cavalcanti, and E. S. De Almeida, "On strategies for testing software product lines: A systematic literature review," Inf. Softw. Technol., vol. 56, no. 10, pp. 1183-1199, 2014.   DOI
8 A. Arrieta, G. Sagardui, and L. Etxeberria, "A model-based testing methodology for the systematic validation of highly configurable cyber-physical systems," in Proc. of 6th International Conference on Advances in System Testing and Validation Lifecycle, VALID 2014, pp. 66-72, 2014.
9 M. Fathi and S. Khoshnevis, "Reusability Metrics in Search-Based Testing of Software Product Lines: An Experimentation," in Proc. of 26th Int. Comput. Conf. Comput. Soc. Iran, CSICC 2021, 2021.
10 A. Saeed, S. H. Ab Hamid, and M. B. Mustafa, "The experimental applications of search-based techniques for model-based testing: Taxonomy and systematic literature review," Appl. Soft Comput. J., vol. 49, pp. 1094-1117, 2016.   DOI
11 M. Bernardino, E. M. Rodrigues, A. F. Zorzo, and L. Marchezan, "Systematic mapping study on MBT: tools and models," IET Softw., vol. 11, no. 4, pp. 141-155, 2017.   DOI
12 M. A. Jamil, M. K. Nour, A. Alhindi, N. S. Awang Abhubakar, M. Arif, and T. F. Aljabri, "Towards Software Product Lines Optimization Using Evolutionary Algorithms," Procedia Comput. Sci., vol. 163, pp. 527-537, 2019.   DOI
13 S. El-Sharkawy, N. Yamagishi-Eichler, and K. Schmid, "Metrics for analyzing variability and its implementation in software product lines: A systematic literature review," Information and Software Technology, vol. 106, pp. 1-30, 2019.   DOI
14 N. Anquetil et al., "Traceability for Model Driven, Software Product Line Engineering," in Proc. of ECMDA Traceability Workshop Proceedings, vol. 12, pp. 77-86, 2008.
15 S. Pradhan, M. Ray, and S. K. Swain, "Transition coverage based test case generation from state chart diagram," J. King Saud Univ. - Comput. Inf. Sci., vol. 34, no. 3, pp. 993-1002, 2022.
16 H. Samih, H. Le Guen, R. Bogusch, M. Acher, and B. Baudry, "Deriving usage model variants for model-based testing: An industrial case study," in Proc. of IEEE Int. Conf. Eng. Complex Comput. Syst. ICECCS, pp. 77-80, 2014.
17 G. Perrouin, S. Oster, S. Sen, J. Klein, B. Baudry, and Y. le Traon, "Pairwise testing for software product lines: Comparison of two approaches," Softw. Qual. J., vol. 20, no. 3-4, pp. 605-643, 2012.   DOI
18 S. Wang and S. Ali, "Modeling bCMS product line using feature model, component family model and UML," in Proc. of CEUR Workshop Proc., vol. 1076, 2013.
19 A. Anjorin, S. Oster, I. Zorcic, and A. Sch, "Optimizing Model-Based Software Product Line Testing with Graph Transformations," in Proc. of Electron. Commun. EASST 11th Int. Work. Graph Transform. Vis. Model. Tech. ( GTVMT 2012 ), vol. 47, 2012.
20 A. O. Elfaki, S. Phon-Amnuaisuk, and C. K. Ho, "Modeling variability in software product line using first order logic," in Proc. of 7th ACIS International Conference on Software Engineering Research, Management and Applications, SERA09, pp. 227-233, 2009.
21 H. Lackner and M. Schmidt, "Towards the assessment of software product line tests: a mutation system for variable systems," in Proc. of 18th Int. Softw. Prod. Line Conf. Companion Vol. Work. Demonstr. Tools - SPLC '14, vol. 2, pp. 62-69, 2014.
22 C. Henard, "Enabling Testing of Large Scale Highly Configurable Systems with Search-based Software Engineering : The Case of Model-based Software Product Lines," Dissertation Defense Committee, Universite Du Luxemborg, 2015.
23 A. Reuys, E. Kamsties, K. Pohl, and S. Reis, "Model-Based System Testing of Software Product Families," in Proc. of Int. Conf. Adv. Inf. Syst. Eng., pp. 519-534, 2005.
24 S. Wang, S. Ali, and A. Gotlieb, "Cost-effective test suite minimization in product lines using search techniques," J. Syst. Softw., vol. 103, pp. 370-391, 2015.   DOI
25 M. Farrag, "Colored Model Based Testing for Software Product Lines (CMBT-SWPL)," Technical University of Ilmenau, 2010.
26 R. A. Sulaiman, D. N. A. Jawawi, and S. A. Halim, "Derivation of test cases for model-based testing of software product line with hybrid heuristic approach," Adv. Intell. Syst. Comput., vol. 1073, pp. 199-208, 2020.   DOI
27 D. Sundmark, P. Wallin, and C. Amlinger, "Model-Based Product Line Engineering in an Industrial Automotive Context : An Exploratory Case Study," in Proc. of the 22nd International Conference on Systems and Software Product Line, vol. v2, pp. 56-63, 2018.
28 S. Wang, S. Ali, and A. Gotlieb, "Automated product line methodologies to support model-based testing," in Proc. of CEUR Workshop Proc., vol. 1115, pp. 56-60, 2013.
29 F. Damiani, D. Faitelson, C. Gladisch, and S. Tyszberowicz, "A novel model-based testing approach for software product lines," Softw. Syst. Model., vol. 16, no. 4, pp. 1223-1251, 2017.   DOI
30 R. M. Hierons, M. Li, X. Liu, J. A. Parejo, S. Segura, and X. Yao, "Many-objective test suite generation for software product lines," ACM Trans. Softw. Eng. Methodol., vol. 29, no. 1, pp. 1-46, 2020.
31 A. Arrieta, S. Wang, G. Sagardui, and L. Etxeberria, "Test Case Prioritization of Configurable Cyber-Physical Systems with Weight-Based Search Algorithms," in Proc. of 2016 Genet. Evol. Comput. Conf. - GECCO '16, pp. 1053-1060, 2016.
32 H. Baller, S. Lity, M. Lochau, and I. Schaefer, "Multi-objective test suite optimization for incremental product family testing," in Proc. of IEEE 7th Int. Conf. Softw. Testing, Verif. Validation, ICST 2014, pp. 303-312, 2014.
33 H. Samih et al., "An Approach to Derive Usage Models Variants for Model-based Testing," in Proc. of The 26th IFIP International Conference on Testing Software and Systems, pp. 80-96, 2016.
34 S. Lity, S. Nahrendorf, T. Thum, C. Seidl, and I. Schaefer, "175 % Modeling for Product-Line Evolution of Domain Artifacts," ACM Comput. Mach., no. 1, pp. 27-34, 2018.
35 A. Knapp and M. Roggenbach, "On the Use of Test Cases in Model-Based Software Product Line Development," in Proc. of 18th Int. Softw. Prod. Line Conf. 1. ACM, no. 3, pp. 247-251, 2014.
36 S. Wang, S. Ali, T. Yue, and M. Liaaen, "Using feature model to support model-based testing of product lines: An industrial case study," in Proc. of 2013 13th International Conference on Quality Software, pp. 75-84, 2013.
37 S. Oster, "Feature Model-based Software Product Line Testing," Ph.D. thesis, Technische Universitat, 2012.
38 I. Schaefer, M. Al-Hajjaji, R. Lachmann, F. Furchtegott, and S. Lity, "Fine-grained test case prioritization for integration testing of delta-oriented software product lines," in Proc. of the 7th International Workshop on Feature-Oriented Software Development, vol. 1, no. 212, pp. 1-10, 2016.
39 I.Machado, "Fault Model-Based Variability Testing," Ph.D. Thesis, Universidade Salvador, 2014.
40 C. D. N. Damasceno, M. R. Mousavi, and A. da S. Simao, "Learning by sampling: learning behavioral family models from software product lines," Empir. Softw. Eng., vol. 26, no. 1, 2021.
41 X. Devroey, G. Perrouin, M. Cordy, H. Samih, A. Legay, and P. S. Patrick, "Statistical prioritization for software product line testing : an experience report," Softw. Syst. Model., vol. 16, no. 1, pp. 153-171, 2017.   DOI
42 P. de A. dos Santos Neto, R. Britto, R. de A. L. Rabelo, J. J. de A. Cruz, and W. A. L. Lira, "A hybrid approach to suggest software product line portfolios," Appl. Soft Comput. J., vol. 49, pp. 1243-1255, 2016.   DOI
43 M. A. Isa, S. A. Razak, D. Norhayati, A. Jawawi, and O. L. Fuh, "Model-Based Testing for Software Product Line : A Systematic Literature Review," Int. J. Softw. Eng. Technol., vol. 02, no. 2, pp. 27-34, 2017.
44 Devroey, X., Perrouin, G., Legay, A., Schobbens, P. Y., and Heymans, P., "Search-based Similarity-driven Behavioural SPL Testing," in Proc. of the Tenth International Workshop on Variability Modelling of Software-intensive Systems, pp. 89-96, 2016.
45 M. Lochau, S. Oster, U. Goltz, and A. Schurr, "Model-based pairwise testing for feature interaction coverage in software product line engineering," Softw. Qual. J., vol. 20, no. 3-4, pp. 567-604, 2012.   DOI
46 E. Kamsties, K. Pohl, S. Reis, and A. Reuys, "Testing Variabilities in Use Case Models," in Proc. of Int. Work. Softw. Prod. Eng., pp. 6-18, 2010.
47 G. Kaur and A. V. Rajoriya, "Approaches to Face Challenges in Product Line Model Based Testing," Int. J. Comput. Syst., vol. 03, no. 03, pp. 207-210, 2016.
48 J. Simmonds and M. C. Bastarrica, "Modeling variability in software process lines," 2011.
49 P. Ardimento, N. Boffoli, and G. Superbo, "Multi software product lines: A systematic mapping study," in Proc. of ENASE 2020 - Proc. 15th Int. Conf. Eval. Nov. Approaches to Softw. Eng., no. Enase, pp. 470-476, 2020.
50 M. Marques, J. Simmonds, P. O. Rossel, and M. C. Bastarrica, "Software product line evolution: A systematic literature review," Inf. Softw. Technol., vol. 105, no. January, pp. 190-208, 2019.   DOI
51 B. Kitchenham, O. Pearl Brereton, D. Budgen, M. Turner, J. Bailey, and S. Linkman, "Systematic literature reviews in software engineering - A systematic literature review," Inf. Softw. Technol., vol. 51, no. 1, pp. 7-15, 2009.   DOI
52 H. Cichos, S. Oster, M. Lochau, and A. Schuerr, "Model-Based Coverage-Driven Test Suite Generation for Software Product Lines," Model Driven Eng. Lang. Syst., vol. 6981, pp. 425-439, 2011.   DOI
53 S. Weissleder and H. Lackner, "Top-Down and Bottom-Up Approach for Model-Based Testing of Product Lines," Electron. Proc. Theor. Comput. Sci., vol. 111, no. Mbt, pp. 82-94, 2013.   DOI
54 M. Lochau, I. Schaefer, J. Kamischke, and S. Lity, "Incremental model-based testing of delta-oriented software product lines," Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol. 7305 LNCS, pp. 67-82, 2012.
55 E. Cartaxo and P. D. L. Machado, "Test case generation by means of UML sequence diagrams and Label Transition System," in Proc. of 2007 IEEE International Conference on Systems, Man and Cybernetics, pp. 1292-1297, 2007.
56 C. S. Gebizli and H. Sozer, "Model-Based Software Product Line Testing by Coupling Feature Models with Hierarchical Markov Chain Usage Models," in Proc. of 2016 IEEE Int. Conf. Softw. Qual. Reliab. Secur. QRS-C 2016, pp. 278-283, 2016.
57 F. Sadia, M. Hasan, N. Nahar, and M. Rokonuzzaman, "A New Process Model of Incremental Asset Building for Software Project Management," in Proc. of 19th Int. Conf. Softw. Eng. Res. Manag. Appl., pp. 86-90, 2021.
58 M. Varshosaz and G. Schneider, "Test Models and Algorithms for Model-Based Testing of Software Product Lines," no. 30. 2017.
59 S. Weissleder, D. Sokenou, and B.-H. Schlingloff, "Reusing State Machines for Automatic Test Generation in Product Lines," in Proc. of 1st Work. Model. Test. Pract. (MoTiP '08), vol. 6, p. 10, 2008.
60 P. Arcaini, O. Inverso, and C. Trubiani, "Automated model-based performance analysis of software product lines under uncertainty," in Proc. of the 25th ACM International Systems and Software Product Line Conference - Volume A, p. 112, 2021.
61 W. K. G. Assuncao, S. R. Vergilio, and R. E. Lopez-Herrejon, "Automatic extraction of product line architecture and feature models from UML class diagram variants," Inf. Softw. Technol., vol. 117, no. March 2019, 2020.   DOI
62 S. J. Prowell, "Using Markov Chain Usage Models to Test Complex Systems," in Proc. of 38th Annu. Hawaii Int. Conf. Syst. Sci., vol. 00, no. C, pp. 318c-318c, 2005.
63 F. Damiani, M. Lienhardt, and L. Paolini, "A formal model for Multi Software Product Lines," Sci. Comput. Program., vol. 172, no. 644298, pp. 203-231, 2019.   DOI
64 D. OZTuRK, "A MODEL-BASED TEST GENERATION APPROACH FOR AGILE SOFTWARE PRODUCT," Doctoral dissertation, Izmir Institute of Technology (Turkey), 2020.
65 M. Utting, A. Pretschner, and B. Legeard, "A taxonomy of model-based testing approaches," Softw. Test. Verif. Reliab., vol. 22, no. 5, pp. 297-312, 2012.   DOI
66 B. Garcia, R. Garcia-carmona, A. Navas, H. A. Parada-gelvez, F. Cuadrado, and J. C. Duenas, "An automated Model-based Testing Approach in Software Product Lines Using a Variability Language," 2010.
67 J. Font, L. Arcega, O. Haugen, and C. Cetina, "Leveraging variability modeling to address metamodel revisions in Model-based Software Product Lines," Comput. Lang. Syst. Struct., vol. 48, pp. 20-38, 2017.   DOI
68 E. D. . Rodrigues, "PLETS - A PRODUCT LINE OF MODEL-BASED TESTING TOOLS," 2013.
69 C. Kastner, A. Von Rhein, S. Erdweg, S. Apel, and K. Ostermann, "Toward Variability-Aware Testing," in Proc. 4th Int. Work. Featur. Softw. Dev., ACM, pp. 1-8, 2012.
70 H. Lackner, M. Thomas, F. Wartenberg, and S. Weissleder, "Model-based test design of product lines: Raising test design to the product line level," in Proc. of IEEE 7th Int. Conf. Softw. Testing, Verif. Validation, ICST 2014, pp. 51-60, 2014.
71 S. Wang, A. Gotlieb, S. Ali, and M. Liaaen, "Automated Test Case Selection Using Feature Model: An Industrial Case Study," Model-Driven Engineering Languages and Systems, pp. 237-253, 2013.
72 F. Ensan, E. Bagheri, and D. Gasevic, "Evolutionary Search-based Test Generation for Software Product Line Feature Models," in Proc. of International Conference on Advanced Information Systems Engineering, pp. 613-628, 2012.
73 G. Perrouin et al., "Automatic and Scalable T-wise Test Case Generation Strategies for Software Product Lines," in Proc. of Third international conference on software testing, verification and validation IEEE, pp. 459-468, 2010.
74 F. Belli, C. J. Budnik, A. Hollmann, T. Tuglular, and W. E. Wong, "Model-based mutation testing-Approach and case studies," Sci. Comput. Program., vol. 120, pp. 25-48, 2016.   DOI
75 K. L. Petry, E. OliveiraJr, and A. F. Zorzo, "Model-based testing of software product lines: Mapping study and research roadmap," J. Syst. Softw., vol. 167, p. 110608, 2020.   DOI
76 T. Buchmann, A. Dotor, and B. Westfechtel, "MOD2-SCM: A model-driven product line for software configuration management systems," Inf. Softw. Technol., vol. 55, no. 3, pp. 630-650, 2013.   DOI
77 S. Wang, A. Gotlieb, S. Ali, and M. Liaaen, "Automated Selection of Test Cases using Feature Model for Product Lines : An Industrial Case Study," Model-Driven Engineering Languages and Systems, pp. 237-253, 2013.
78 S. Wang, "Systematic Product Line Testing : Methodologies, Automation, and Industrial Application," PhD Thesis, University of Oslo (UiO), 2015.
79 K. Petry, E. OliveiraJr, L. Costa, A. Zanin, and A. Zorzo, "SMartyTesting: A Model-Based Testing Approach for Deriving Software Product Line Test Sequences," in Proc. of Int. Conf. Enterp. Inf. Syst., vol. 2, no. Iceis, pp. 165-172, 2021.
80 S. Oster, Feature Model-based Software Product Line Testing, 2012.
81 S. Wang and S. Ali, "Minimizing Test Suites in Software Product Lines Using Weight-based Genetic Algorithms," in Proc. of 15th Annu. Conf. Genet. Evol. Comput, ACM, pp. 1493-1500, 2013.
82 R. A. Sulaiman, D. N. A. Jawawi, and S. A. Halim, "A dissimilarity with dice-jaro-winkler test case prioritization approach for model-based testing in software product line," KSII Trans. Internet Inf. Syst., vol. 15, no. 3, pp. 932-951, 2021.
83 C. Henard, M. Papadakis, G. Perrouin, J. Klein, and Y. Le Traon, "Assessing software product line testing via model-based mutation: An application to similarity testing," in Proc. of IEEE 6th Int. Conf. Softw. Testing, Verif. Valid. Work. ICSTW 2013, pp. 188-197, 2013.
84 C. Henard, M. Papadakis, G. Perrouin, and J. Klein, "Multi-objective Test Generation for Software Product Lines," in Proc. of 17th Int. Softw. Prod. Line Conf. ACM, pp. 62-71, 2013.
85 M. Pol'la, A. Buccella, and A. Cechich, "Analysis of variability models: a systematic literature review," Softw. Syst. Model., vol. 20, no. 4, pp. 1043-1077, 2021.   DOI
86 H. Yadav, A. Charan Kumari, and R. Chhikara, "Feature selection optimisation of software product line using metaheuristic techniques," Int. J. Embed. Syst., vol. 13, no. 1, pp. 50-64, 2020.   DOI
87 H. Lackner, "Model-based product line testing: Sampling configurations for optimal fault detection," Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol. 9369, pp. 238-251, 2015.
88 M. Al-Hajjaji, "Scalable and Efficient Sampling for Product-Line Testing," Tech. Rep. FIN-003-2014, Univ. Magdeburg, Ger. 2014, 2014.
89 R. A. Sulaiman, D. N. A. Jawawi, and S. A. Halim, "Coverage-based approach for model-based testing in Software Product Line," Int. J. Eng. Technol., vol. 7, no. 4, pp. 63-68, 2018.
90 X. Devroey, G. Perrouin, A. Legay, M. Cordy, P. Schobbens, and P. Heymans, "Coverage Criteria for Behavioural Testing of Software Product Lines," in Proc. of 6th Int. Symp. Leveraging Appl. Form. Methods, Verif. Valid. (to Appear.), pp. 336-350, 2014.
91 R. Sulaiman, D. N. A. Jawawi, and S. A. Halim, "Features and Behaviours Mapping in Model-based Testing in Software Product Line," in Proc. of IOP Conf. Ser. Mater. Sci. Eng., vol. 884, no. 1, 2020.
92 J. S. Her, J. H. Kim, S. H. Oh, S. Y. Rhew, and S. D. Kim, "A framework for evaluating reusability of core asset in product line engineering," Inf. Softw. Technol., vol. 49, no. 7, pp. 740-760, 2007.   DOI
93 X. Lian, L. Zhang, J. Jiang, and W. Goss, "An Approach for Optimized Feature Selection in Large-scale Software Product Lines," J. Syst. Softw., vol. 137, pp. 636-651, 2018.   DOI
94 R. A. Matnei Filho and S. R. Vergilio, "A multi-objective test data generation approach for mutation testing of feature models," J. Softw. Eng. Res. Dev., vol. 4, no. 1, p. 4, 2016.   DOI
95 D. Reuling, J. Burdek, S. Rotarmel, M. Lochau, and U. Kelter, "Fault-based product-line testing," in Proc. of SPLC - Int. Conf. Softw. Prod. lines, pp. 131-140, 2015.
96 S. Wang, S. Ali, A. Gotlieb, and M. Liaaen, "A systematic test case selection methodology for product lines: results and insights from an industrial case study," Empir. Softw. Eng., vol. 21, no. 4, pp. 1586-1622, 2016.   DOI