Journal of Computing Science and Engineering

Korean Institute of Information Scientists and Engineers (한국정보과학회)
권호 표지
DOI QR코드

DOI QR Code

An Empirical Evaluation of Test Data Generation Techniques

  • Published : 2008.09.30
Download PDF
⟨ Previous Next ⟩

Abstract

Software testing cost can be reduced if the process of testing is automated. However, the test data generation task is still performed mostly by hand although numerous theoretical works have been proposed to automate the process of generating test data and even commercial test data generators appeared on the market. Despite prolific research reports, few attempts have been made to evaluate and characterize those techniques. Therefore, a lot of works have been proposed to automate the process of generating test data. However, there is no overall evaluation and comparison of these techniques. Evaluation and comparison of existing techniques are useful for choosing appropriate approaches for particular applications, and also provide insights into the strengths and weaknesses of current methods. This paper conducts experiments on four representative test data generation techniques and discusses the experimental results. The results of the experiments show that the genetic algorithm (GA)-based test data generation performs the best. However, there are still some weaknesses in the GA-based method. Therefore, we modify the standard GA-based method to cope with these weaknesses. The experiments are carried out to compare the standard GA-based method and two modified versions of the GA-based method.

Keywords

References

  1. ANDREOU, A. S., ECONOMIDES, K. A., AND SOFOKLEOUS, A. A. 2007. An automatic software test-data generation scheme based on data flow criteria and genetic algorithms. In Proceedings of the 7th IEEE International Conference on Computer and Information Technology, 867-872.
  2. BEIZER, B. 1990. Software Testing Techniques(2nd edition). Van Nostrand Reinhold.
  3. BODIK, R., GUPTA, R., AND SOFFA, M. L. 1997. Refining data flow information using infeasible paths. In Proceedings of the Sixth European Software Engineering Conference (ESEC/ FSE 97), M. Jazayeri and H. Schauer, Eds. Lecture Notes in Computer Science 1013, Springer-Verlag, 361-377.
  4. CLARKE, L. A. 1976. A system to generate test data and symbolically execute programs. IEEE Transactions on Software Engineering, 2(3):215-222.
  5. COWARD, P. D. 1988. Symbolic execution systems-a review. Software Engineering Journal 3, 6, 229-239. https://doi.org/10.1049/sej.1988.0029
  6. EDVARDSSON, J. 1999. A survey on automatic test data generation. In Proceedings of the 2nd Conference on Computer Science and Engineering in Linkoping, 21-28.
  7. FERGUSON, R. AND KOREL, B. 1995. Software test data generation using the chaining approach. In ITC. IEEE Computer Society, 703-709.
  8. GALLAGHER, M. J. AND NARASIMHAN, V. L. 1997. ADTEST: A test data generation suite for ada software systems. IEEE Transactions on Software Engineering, 23(8):473-484. https://doi.org/10.1109/32.624304
  9. GEN, M. AND CHENG, R. 1997. Genetic Algorithms and Engineering Design. John Wiley and Sons, Inc., New York.
  10. GUPTA, N., CHO, Y., AND HOSSAIN, M. Z. 2004. Experiments with una for solving linear constraints in real variables. In Proceedings of the 2004 ACM symposium on Applied computing. ACM, 1013-1020.
  11. GUPTA, N., MATHUR, A. P., AND SOFFA, M. L. 1998. Automated test data generation using an iterative relaxation method. In Proceedings of the ACM SIGSOFT 6th International Symposium on the Foundations of Software Engineering (FSE-98). Software Engineering Notes, 23(6), ACM Press, 231-244.
  12. GUPTA, N., MATHUR, A. P., AND SOFFA, M. L. 1999. UNA based iterative test data generation and its evaluation. In 14th IEEE International Conference on Automated Software Engineering. IEEE Computer Society Press, 224-234.
  13. GUPTA, N., MATHUR, A. P., AND SOFFA, M. L. 2000. Generating test data for branch coverage. In Proceedings of IEEE International Conference on ASE, 219-228.
  14. KOREL, B. 1990a. Automated software test data generation. IEEE Transactions on Software Engineering, 16(8):870-879. https://doi.org/10.1109/32.57624
  15. KOREL, B. 1990b. A dynamic approach of test data generation. In Conference on Software Maintenance, 311-317.
  16. LAPIERRE, S., MERLO, E., SAVARD, G., ANTONIOL, G., FIUTEM, R., AND TONELLA, P. 1999a. Automatic unit test data generation using mixed-integer linear programming and execution trees. In Proceedings of IEEE International Conference on Software Maintenance. IEEE Computer Society Press, 189-198.
  17. LAPIERRE, S., MERLO, E., SAVARD, G., ANTONIOL, G., FIUTEM, R., AND TONELLA, P. 1999b. Automatic unit test data generation using mixed-integer linear programming and execution trees. In Proceedings of IEEE International Conference on Software Maintenance(ICSM '99).
  18. MCCABE, T. J. 1976. A complexity measure. IEEE Transactions on Software Engineering, 2(4): 308-320.
  19. MEUDEC, C. 2001. ATGen: automatic test data generation using constraint logic programming and symbolic execution. Software Testing, Verification and Reliability, 11(2):81-96. https://doi.org/10.1002/stvr.225
  20. MICHAEL, C. C., MCGRAW, G., AND SCHATZ, M. 2001. Generating software test data by evolution. IEEE Trans. Software Eng., 27(12):1085-1110. https://doi.org/10.1109/32.988709
  21. OFFUTT, A. J., LEE, A., ROTHERMEL, G., UNTCH, R. H., AND ZAPF, C. 1996. An experimental determination of sufficient mutant operators. ACM Transactions on Software Engineering and Methodology, 5(2):99-118. https://doi.org/10.1145/227607.227610
  22. PARGAS, R. P., HARROLD, M. J., AND PECK, R. 1999. Test-data generation using genetic algorithms. Software Testing, Verification and Reliability, 9(4):263-282. https://doi.org/10.1002/(SICI)1099-1689(199912)9:4<263::AID-STVR190>3.0.CO;2-Y
  23. SCHEID, F. 1968. Numerical Analysis, Schaum's Outline Series. McGraw-Hill Book Company.
  24. STHAMER, H. H. 1995. The automatic generation of software test data using genetic algorithms. Ph.D. thesis, University of Glamorgan.