KIPS Transactions on Software and Data Engineering (정보처리학회논문지:소프트웨어 및 데이터공학)

Korea Information Processing Society (한국정보처리학회)
권호 표지
DOI QR코드

DOI QR Code

Code Coverage Measurement in Configurable Software Product Line Testing

구성가능한 소프트웨어 제품라인 시험에서 코드 커버리지 측정

  • 한수빈 (전북대학교 소프트웨어공학과) ;
  • 이지현 (전북대학교 소프트웨어공학과) ;
  • 고서연 (전북대학교 소프트웨어공학과)
  • Received : 2021.12.28
  • Accepted : 2022.02.28
  • Published : 2022.07.31
Download PDF
⟨ Previous Next ⟩

Abstract

Testing approaches for configurable software product lines differs significantly from a single software testing, as it requires consideration of common parts used by all member products of a product line and variable parts shared by some or a single product. Test coverage is a measure of the adequacy of testing performed. Test coverage measurements are important to evaluate the adequacy of testing at the software product line level, as there can be hundreds of member products produced from configurable software product lines. This paper proposes a method for measuring code coverage at the product line level in configurable software product lines. The proposed method tests the member products of a product line after hierarchizing member products based on the inclusion relationship of the selected features, and quantifies SPL(Software Product Line) test coverage by synthesizing the test coverage of each product. As a result of applying the proposed method to 11 configurable software product line cases, we confirmed that the proposed method could quantitatively visualize how thoroughly the SPL testing was performed to help verify the adequacy of the SPL testing. In addition, we could check whether the newly performed testing for a member product covers the newly added code parts of a feature.

구성가능한 소프트웨어 제품라인 시험은 모든 멤버제품들에 포함되는 공통 부분과 일부 또는 단일 제품에 포함되는 가변 부분을 고려해야 하기 때문에 단일 제품 시험 방법들을 그대로 적용하기 어렵다. 소프트웨어 시스템 시험에서 시험 커버리지는 수행된 시험의 적절성을 측정하는 척도이다. 구성가능한 소프트웨어 제품라인으로부터 생성될 수 있는 멤버제품들은 수백 개에 이를 수 있기 때문에 시험 커버리지 측정은 소프트웨어 제품라인 수준에서 시험의 적절성을 평가하기 위해 중요하다. 이 논문은 구성가능한 소프트웨어 제품라인 시험에서 제품라인 수준의 코드 커버리지 측정 방법을 제안한다. 제안한 방법은 피처 집합의 포함 관계를 기준으로 제품들을 계층화한 후 제품라인의 멤버제품들을 시험하고, 시험으로부터 얻어진 각 제품의 시험 커버리지들을 종합하여 SPL 시험 커버리지를 정량화한다. 제안한 방법을 11개의 구성가능한 소프트웨어 제품라인 사례에 적용하여 검증한 결과, 제안한 방법은 SPL 시험이 얼마나 철저하게 수행되었는지를 정량적으로 기술하여 SPL 시험의 적정성을 확인할 수 있도록 도왔다. 또한, 새로 추가된 SPL 멤버제품의 시험이 이전 멤버제품 시험 대비 커버리지를 증가시키는 방향인지 확인할 수 있어 새로운 시험케이스들이 멤버제품들 간의 차이를 커버하는 방향으로 시험이 이루어지고 있는지 확인할 수 있었다.

Keywords

Acknowledgement

이 성과는 정부(과학기술정보통신부)의 재원으로 한국연구재단의 지원을 받아 수행된 연구임(No.NRF-2020R1F1A1071650).

References

  1. K. Pohl, G. Bockle, and F. van der Linden, "Software product line engineering: Foundations, principles, and techniques," Springer, 2005.
  2. P. M. Clements and L. M. Northrop, "Software product lines: Practices and patterns," Addison Wesley, 2002.
  3. J. Lee, S. Kang, and D. Lee, "A survey on software product line testing," in Proceedings of the 16th International Software Product Line Conference (SPLC'12), Salvador, Brazil, pp.2-7, Sep. 2012.
  4. J. Lee, S. Kang, and P. Jung, "Test coverage criteria for software product line testing: Systematic literature review," Information and Software Technology, Vol.122, pp.1-22, 2020.
  5. P. Jung, S. Kang, and J. Lee, "Efficient regression testing of software product lines by reducing redundant test executions," Applied Sciences, Vol.10, No.23, pp.1-21, 2020.
  6. F. Ferreira, M. Viggiato, M. Souza, and E. Figueiredo, "Testing configurable software systems: The failure observation challenge," in Proceedings of the ACM Software and Systems Product Line Conference (SPLC'20), pp.1-6, Oct. 2020.
  7. S. Kang, J. Lee, Y. Han, and P. Jung, "Introduction to software product line development," HongReung Publishing, Company, 2021.
  8. M. B. Cohen, M. B. Dwyer, and J. Shi, "Constructing interaction test suites for highly-configurable systems in the presence of constraints: A greedy approach," IEEE Transactions On Software Engineering, Vol.34, No.5, pp.633-650, 2008. https://doi.org/10.1109/TSE.2008.50
  9. S. Oster, F. Markert, and P. Ritter, "Automated incremental pairwise testing of software product lines," in Proceedings of the ACM Software and Systems Product Line Conference (SPLC'10), pp.196-210, Sep. 2010.
  10. G. Perrouin, S. Sen, J. Klein, B. Baudry, and Y. Traon, "Automated and scalable t-wise test case generation strategies for software product lines," in Proceedings of the 3rd International Conference on Software Testing, Verification and Validation (ICST'10), pp.459-468, Apr. 2010.
  11. C. H. P. Kim, D. S. Batory, and S. Khurshid, "Reducing combinatorics in testing product lines," in Proceedings of the 10th International Conference on Aspect-oriented Software Development (AOSD'11), pp.57-68, Mar. 2011.
  12. M. Kowal, S. Schulze, and I. Schaefer, "Towards efficient SPL testing by variant reduction," in Proceedings of the 4th International Workshop on Variability & Composition (VariComp'13), pp.1-6, Mar. 2013.
  13. D. Shimbara and O. Haugen, "Generating configurations for system testing with common variability language," Lecture Notes in Computer Science, Vol.9369, pp.221-237, 2015.
  14. M. Al-Hajjaji, S. Krieter, T. Thum, M. Lochau, and G. Saake, "IncLing: Efficient product-line testing using incremental pairwise sampling," in Proceedings of the 2016 ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences (GPCE'16), pp.144-155, Oct. 2016.
  15. A. Strickler, J. A. P. Lima, S. R. Vergilio, and A. T. R. Pozo, "Deriving products for variability test of feature models with a hyper-heuristic approach," Applied Soft Computing, Vol.49, pp.1232-1242, 2016. https://doi.org/10.1016/j.asoc.2016.07.059
  16. A. Hervieua, D. Marijanb, A. Gotliebb, and B. Baudry, "Practical minimization of pairwise-covering test configurations using constraint programming," Information and Software Technology, Vol.71, pp.129-146, 2016. https://doi.org/10.1016/j.infsof.2015.11.007
  17. M. Al-Hajjaji, J. Kruger, S. Schulze, T. Leich. and G. Saake, "Efficient product-line testing using cluster-based product prioritization," in Proceedings of the IEEE/ACM 12th International Workshop on Automation of Software Testing (AST'17), pp.16-22, May 2017.
  18. J. Shi, M. Cohen, and M. Dwyer, "Integration testing of software product lines using compositional symbolic execution," in Proceedings of the 15th International Conference on Fundamental Approaches to Software Engineering (FASE'12), pp.270-284, 2012.
  19. M. Al-Hajjaji, T. Thum, J. Meinicke, M. Lochau, and G. Saake, "Similarity-based prioritization in software product-line testing," in Proceedings of the 16th International Software Product Line Conference (SPLC'14), Sep. pp.197-206, 2014.
  20. J. A. Galindo, H. Turner, D. Benavides, and J. White, "Testing variability-intensive systems using automated analysis: An application to android," Software Quality Journal, Vol.24, pp.365-405, 2016. https://doi.org/10.1007/s11219-014-9258-y
  21. P. Ammann and J. Offutt, "Introduction to Software Testing," 2nd Ed., Cambridge University Press, 2016.
  22. L. Vidacs, F. Horvath, J. Mithalicza, B. Vancsics, and A. Beszedes, "Supporting software product line testing by optimizing code configuration coverage," in Proceedings of the IEEE 8th International Conference on Software Testing, Verification and Validation Workshops (ICSTW'15), pp.1-7, 2015.
  23. E. Uzuncaova, S. Khurshid, and D. Batory, "Incremental test generation for software product lines," IEEE Transactions on Software Engineering, Vol.36, No.3, pp.309-322, 2010. https://doi.org/10.1109/TSE.2010.30
  24. S. Apel, D. Batory, C. Kastner, and G. Saake, "Feature-Oriented Software Product Lines: Concepts and Implementation," Springer, 2013.
  25. Java Code Coverage for Eclipse, [Internet], https://www.jacoco.org, last accessed in 27th Dec. 2021.