Journal of Computing Science and Engineering

Korean Institute of Information Scientists and Engineers (한국정보과학회)
권호 표지
DOI QR코드

DOI QR Code

Selecting Test Cases for Result Inspection to Support Effective Fault Localization

  • Li, Yihan (State Key Laboratory of High-end Server & Storage Technology) ;
  • Chen, Jicheng (State Key Laboratory of High-end Server & Storage Technology) ;
  • Ni, Fan (State Key Laboratory of High-end Server & Storage Technology) ;
  • Zhao, Yaqian (State Key Laboratory of High-end Server & Storage Technology) ;
  • Wang, Hongwei (State Key Laboratory of High-end Server & Storage Technology)
  • Received : 2015.08.30
  • Accepted : 2015.09.07
  • Published : 2015.09.30
Download PDF
⟨ Previous Next ⟩

Abstract

Fault localization techniques help locate faults in source codes by exploiting collected test information and have shown promising results. To precisely locate faults, the techniques require a large number of test cases that sufficiently exercise the executable statements together with the label information of each test case as a failure or a success. However, during the process of software development, developers may not have high-coverage test cases to effectively locate faults. With the test case generation techniques, a large number of test cases without expected outputs can be automatically generated. Whereas the execution results for generated test cases need to be inspected by developers, which brings much manual effort and potentially hampers fault-localization effectiveness. To address this problem, this paper presents a method to select a few test cases from a number of test cases without expected outputs for result inspection, and in the meantime selected test cases can still support effective fault localization. The experimental results show that our approach can significantly reduce the number of test cases that need to be inspected by developers and the effectiveness of fault localization techniques is close to that of whole test cases.

Keywords

References

  1. I. Vessey, "Expertise in debugging computer programs: a process analysis," International Journal of Man-Machine Studies, vol. 23, no. 5, pp. 459-494, 1985. https://doi.org/10.1016/S0020-7373(85)80054-7
  2. J. A. Jones, M. J. Harrold, J. Stasko, "Visualization of test information to assist fault localization," in Proceedings of the 24th International Conference on Software Engineering (ICSE), Orlando, FL, 2002, pp. 467-477.
  3. R. Abreu, P. Zoeteweij, and A. J. van Gemund, "An evaluation of similarity coefficients for software fault localization," in Proceedings of 12th Pacific Rim International Symposium on Dependable Computing (PRDC'06), Riverside, CA, 2006, pp. 39-46.
  4. L. Naish, H. J. Lee, and K. Ramamohanarao, "A model for spectra-based software diagnosis," ACM Transactions on Software Engineering and Methodology, vol. 20, no. 3, article no. 11, 2011.
  5. J. A. Jones and M. J. Harrold, "Empirical evaluation of the Tarantula automatic fault localization technique," in Proceedings of the 20th IEEE/ACM International Conference on Automated Software Engineering, Long Beach, CA, 2005, pp. 273-282.
  6. S. Ali, J. H. Andrews, T. Dhandapani, and W. Wang, "Evaluating the accuracy of fault localization techniques," in Proceedings of the 2009 IEEE/ACM International Conference on Automated Software Engineering, Auckland, New Zealand, 2009, pp. 76-87.
  7. C. Parnin and A. Orso, "Are automated debugging techniques actually helping programmers?," in Proceedings of the 2011 International Symposium on Software Testing and Analysis, Toronto, Canada, 2011, pp. 199-209.
  8. Y. Yu, J. A. Jones, and M. J. Harrold, "An empirical study of the effects of test-suite reduction on fault localization," in Proceedings of the 30th International Conference on Software Engineering, Leipzig, Germany, 2008, pp. 201-210.
  9. P. Godefroid, N. Klarlund, and K. Sen, "DART: directed automated random testing," in Proceedings of the 2005 ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), Chicago, IL, 2005, pp. 213-223.
  10. K. Sen, D. Marinov, and G. Agha, "Cute: a concolic unit testing engine for C," in Proceedings of the 10th European Software Engineering Conference held jointly with 13th ACM SIGSOFT International Symposium on Foundations of Software Engineering, Lisbon, Portugal, 2005, pp. 263-272.
  11. E. F. Barbosa, J. C. Maldonado, and A. M. R. Vincenzi, "Toward the determination of sufficient mutant operators for C," Software Testing, Verification and Reliability, vol. 11, no. 2, pp. 113-136, 2001. https://doi.org/10.1002/stvr.226
  12. X. Xie, W. E. Wong, T. Y. Chen, and B. Xu, "Metamorphic slice: an application in spectrum-based fault localization," Information and Software Technology, vol. 55, no. 5, pp. 866-879, 2013. https://doi.org/10.1016/j.infsof.2012.08.008
  13. A. Gonzalez-Sanchez, E. Piel, H. G. Gross, and A. J. Van Gemund, "Prioritizing tests for software fault localization," in Proceedings of 2010 10th International Conference on Quality Software (QSIC), Zhangjiajie, China, 2010, pp. 42-51.
  14. A. Gonzalez-Sanchez, R. Abreu, H. G. Gross, and A. J. Van Gemund, "Prioritizing tests for fault localization through ambiguity group reduction," in Proceedings of 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE), Lawrence, KS, 2011, pp. 83-92.
  15. L. Gong, D. Lo, L. Jiang, and H. Zhang, "Diversity maximization speedup for fault localization," in 2012 Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering (ASE), Essen, Germany, 2012, pp. 30-39.
  16. S. Artzi, J. Dolby, F. Tip, and M. Pistoia, "Directed test generation for effective fault localization," in Proceedings of the 19th International Symposium on Software Testing and Analysis, Trento, Italy, 2010, pp. 49-60.
  17. D. Hao, T. Xie, L. Zhang, X. Wang, J. Sun, and H. Mei, "Test input reduction for result inspection to facilitate fault localization," Automated Software Engineering, vol. 17, no. 1, pp. 5-31, 2010. https://doi.org/10.1007/s10515-009-0056-x
  18. Y. H. Li and L. L. Yu, "A test case selection method to support fault localization," in Proceedings of 2014 International Conference on Computer Science and Software Engineering (CSSE2014), Hangzhou, China, 2014, pp. 129-135.
  19. B. Baudry, F. Fleurey, and Y. Le Traon, "Improving test suites for efficient fault localization," in Proceedings of the 28th International Conference on Software Engineering, Shanghai, China, 2006, pp. 82-91.
  20. D. Hao, L. Zhang, Y. Pan, H. Mei, and J. Sun, "On similarity-awareness in testing-based fault localization," Automated Software Engineering, vol. 15, no. 2, pp. 207-249, 2008. https://doi.org/10.1007/s10515-008-0025-9
  21. W. Masri, R. Abou-Assi, M. El-Ghali, and N. Al-Fatairi, "An empirical study of the factors that reduce the effectiveness of coverage-based fault localization," in Proceedings of the 2nd International Workshop on Defects in Large Software Systems: held in conjunction with the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2009), Chicago, IL, 2009, pp. 1-5.
  22. C. Liu, X. Zhang, and J. Han, "A systematic study of failure proximity," IEEE Transactions on Software Engineering, vol. 34, no. 6, pp. 826-843, 2008. https://doi.org/10.1109/TSE.2008.66
  23. C. Liu, X. Yan, L. Fei, J. Han, and S. P. Midkiff, "SOBER: statistical model-based bug localization," ACM SIGSOFT Software Engineering Notes, vol. 30, no. 5, pp. 286-295, 2005. https://doi.org/10.1145/1095430.1081753
  24. Z. Zhang, W. K. Chan, T. H. Tse, Y. T. Yu, and P. Hu, "Nonparametric statistical fault localization," Journal of Systems and Software, vol. 84, no. 6, pp. 885-905, 2011. https://doi.org/10.1016/j.jss.2010.12.048