Asia pacific journal of information systems

The Korea Society of Management Information Systems (한국경영정보학회)
권호 표지

Application of Empirical Research Methods in Information Systems Research: Gaining Lessons Through Evaluation

경영정보학 연구에 나타난 실증적 연구방법 적용상의 문제: 평가를 통해 얻은 교훈

  • 강신철 (한남대학교 경영정보학과) ;
  • 이중기 (연세대학교 정보대학원) ;
  • 최정일 (Merrimack College 경영학과)
  • Published : 2006.06.30
Download PDF
⟨ Previous Next ⟩

Abstract

Application of appropriate research methods has assumed an important role in knowledge accumulation endeavors in the scientific research community. The current study reported here summarizes how we used the empirical methods in IS research, what we need to improve in using empirical study as research methodology through the set of comprehensive research guideline. From the survey and review of seminal and classical research guidelines, we developed our own 28 checklist for research design, statistical analysis, and conclusion, which can be commonly applied to all articles that employ inferential statistical methods in IS research. Then, we critically evaluated the usage of empirical research methods in major IS journals by using the checklist, with the goal of improving the quality of academic papers. In this study, we scrutinized four major IS journals which published empirical papers from 1991 to 2000: MIS Quarterly, Journal of MIS, Information Systems Research, and Decision Sciences. As a result of intensive evaluation work, we could highlight many areas that are lagging and call for greater attention with regard to the proper usage of empirical study in IS research. The research findings in this study can be referred as checklist and guideline when IS researcher applies the empirical method.

Keywords

References

  1. Adams, D.A., Nelson, R.R., and Todd, P.A., 'Perceived Usefulness, Ease of Use, and Usage of Information Technology: A Replication,' MIS Quarterly, Vol. 16, No. 2, 1992, pp. 227-247 https://doi.org/10.2307/249577
  2. Agarwal, R. and Prasad, J., 'The Antecedents and Consequents of User Perceptions in Information Technology Adoption,' Decision Support Systems, Vol. 22, No. 2, 1998, pp. 15-29 https://doi.org/10.1016/S0167-9236(97)00006-7
  3. Ajzen, I., 'From Intensions to Actions: A Theory of Planned Behavior. In J. Kuhl & J. Beckmann (Eds.),' Action Control: From Cognition to Behavior (pp. 11-39). New York: Springer, 1985
  4. Ajzen, I., 'The Theory of Planned Behavior,' Organizational Behavior and Human Decision Processes, Vol. 50, No. 2, 1991, pp. 179-211 https://doi.org/10.1016/0749-5978(91)90020-T
  5. Alavi, M. and Carlson, P., 'A Review of MIS Research and Disciplinary Development,' Journal of Management Information Systems, Vol. 8, No. 4, 1992, pp. 45-62 https://doi.org/10.1080/07421222.1992.11517938
  6. Bacharach, S.B., 'Organizational Theories: Some Criteria for Evaluation,' Academy of Management Review, Vol. 14, No. 4, 1989, pp. 496-515 https://doi.org/10.2307/258555
  7. Bailar, J.C., III., and Mosteller, F., 'Guidelines for Statistical Reporting in Articles for Medical Journals,' Annals of Internal Medicine, Vol. 108, 1988, pp. 266-273 https://doi.org/10.7326/0003-4819-108-2-266
  8. Baroudi, J.J. and Olikowski, W.J., 'The Problems of Statistical Power in MIS Research,' MIS Quarterly, Vol. 13, No. 1, 1988, pp. 88-106
  9. Boudreau, M., Gefen, D., and Straub, D. W., 'Validation in IS Research: A State- of-the Art Assessment,' MIS Quarterly, Vol. 25, No. 1, 2001, pp. 1-16 https://doi.org/10.2307/3250956
  10. Bowers, J.W. and Courtright, J.A., Communication Research Methods, Glenview, IL, Scott, Foresman, 1984
  11. Burnett, J.J. and Dunne, P.M., 'An Appraisal of the Use of Student Subjects in Marketing Research,' Journal of Business Research, Vol. 14, No. 4, 1988, pp. 329-343 https://doi.org/10.1016/0148-2963(86)90024-X
  12. Campion, M.A., 'Article Review Checklist: A Criterion Checklist for Reviewing Research Articles in Applied Psychology,' Personnel Psychology, Vol. 46, 1993, pp. 705-718 https://doi.org/10.1111/j.1744-6570.1993.tb00896.x
  13. Chau, P.Y.K., 'On the Use of Construct Reliability in MIS Research: A Meta-Analysis,' Information and Management, Vol. 35, No. 4, 1999, pp. 217-227 https://doi.org/10.1016/S0378-7206(98)00089-5
  14. Chin, W.W. and Todd, P.A., 'On the Use, Usefulness, and Ease of Use of Structural Equation Modeling in MIS research: A Note of Caution,' MIS Quarterly, Vol. 19, No. 2, 1995, pp. 237-246 https://doi.org/10.2307/249690
  15. Cohen, J., 'A Coefficient of Agreement for Nominal Scales,' Educational and Psychological Measurement, Vol. 20, 1960, pp. 37-46 https://doi.org/10.1177/001316446002000104
  16. Cohen, J., Statistical Power Analysis for the Behavioral Sciences (2nd ed.), Hillsdale, NJ: Erlbaum, 1988
  17. Culnan, M. and Swanson, E., 'Research in Management Information Systems, 1980- 1984; Points of Work and Reference,' MIS Quarterly, Vol. 7, No. 3, 1986, pp. 286-303
  18. Davis, F.D., 'Perceived Usefulness, Perceived Ease of Use and User Acceptance of Information Technology,' MIS Quarterly, Vol. 13, No. 3, 1989, pp. 319-339 https://doi.org/10.2307/249008
  19. Delucchi, K.L., 'The Use and Misuse of Chi-Square: Lewis and Burke Revisited,' Psychological Bulletin, Vol. 94, No. 1, pp. 1987, 166-176 https://doi.org/10.1037/0033-2909.94.1.166
  20. Gaither, N. and Glorfeld, L., 'An Evaluation of the Use of Tests of Significance in Organizational Behavior Research,' Academy of Management Review, Vol. 10, No. 4, 1985, pp. 787-793 https://doi.org/10.2307/258046
  21. Gardner, W., On the Reliability of Sequential Data: Measurement, Meaning and Correction, Mahwah, NJ: Erlbaum, 1995
  22. Hair, J.F., Anderson, R.E., Tatham, R.L., and Black, W.C., Multivariate Data Analysis (5th ed.), Upper Saddle River, NJ: Prentice Hall, 1998
  23. Hamilton, S. and Ives, B., 'MIS Research Strategies,' Information and Management, Vol. 5, No. 4, 1982, pp. 339-347 https://doi.org/10.1016/0378-7206(82)90033-7
  24. Hughes, C.T. and Gibson, M.L., 'Students as Surrogates for Managers in a Decision- making Environment: An Experimental Study,' Journal of Management Information Systems, Vol. 8, No. 2, 1991, pp. 153-166 https://doi.org/10.1080/07421222.1991.11517925
  25. Jarvenpaa, S., 'The Effect of Task Demands and Graphical Format on Information Processing Strategies,' Management Science, Vol. 35, No. 3, 1989, pp. 285-303 https://doi.org/10.1287/mnsc.35.3.285
  26. Jarvenpaa, S., Dickson, G., and DeSanctis, G., 'Methodological Issues in Experimental IS Research: Experiences and Recommendations,' MIS Quarterly, Vol. 9, No. 2, 1985, pp. 237-246
  27. Keen, P., MIS Research: Reference Disciplines and a Cumulative Tradition, Paper presented at the The First International Conference on Information Systems, Philadelphia, PA, 1980
  28. Kerlinger, F.N., Foundations of Behavioral Research (3rd ed.), New York: CBS College Publishing, 1986
  29. Kraemer, K.L. and Dutton, W.H., 'Survey Research in the Study of Management Information Systems,' In K. L. Kraemer (Ed.), The Information Systems Research Challenges: Survey Research Methods (Vol. 3, pp. 3-57), Cambridge, MA: Harvard Business School Press, 1991
  30. Landis, J. and Koch, G.G., 'The Measurement of Observer Agreement for Categorical Data,' Biometrics, Vol. 33, 1977, pp. 159-174 https://doi.org/10.2307/2529310
  31. LaTour, S.A. and Miniard, P.W., 'The Misuse of Repeated Measures Analysis in Marketing Research,' Journal of Mar Marketing Research,' Journal of Marketing Research, Vol. 20, No. 1, 1983, pp. 45-47 https://doi.org/10.2307/3151411
  32. Lee, A.S., 'A Scientific Methodology for MIS Case Studies,' MIS Quarterly, Vol. 13, No. 1, 1989, pp. 33-49 https://doi.org/10.2307/248698
  33. Lucas, H.C., 'Methodological Issues in Information Systems Survey Research,' In K. L. Kraemer (Ed.), The Information Systems Research Challenges: Survey Research Methods (Vol. 3, pp. 273-282). Cambridge, MA: Harvard Business School Press, 1991
  34. Marx, M.H., 'Formal Theory,' In M. H. Marx and F. E. Goodson (Eds.), Theories in Contemporary Psychology (2nd ed., pp. 234-260), New York: Macmillan, 1976a
  35. Marx, M.H., 'Theorizing,' In M. H. Marx and F. E. Goodson (Eds.), Theories in Contemporary Psychology (2nd ed., pp. 261- 286), New York: Macmillan, 1976b
  36. Meyers, L.S. and Grossen, N.E., Behavioral Research: Theory, Procedure, and Design, San Francisco, CA: W.H. Freeman and Company, 1974
  37. Miles, M.B. and Huberman, A.M., Qualitative Data Analysis: An Expanded Sourcebook, Thousand Oaks, CA: Sage Publications, Inc, 1994
  38. Mitchell, T.R., 'An Evaluation of the Validity of Correlational Research Conducted in Organizations,' Academy of Management Review, Vol. 10, No. 2, 1985, pp. 192-205 https://doi.org/10.2307/257962
  39. Moore, G.C. and Benbasat, I., 'Development of an Instrument to Measure the Perceptions of Adopting an Information Technology Innovation,' Information Systems Research, Vol. 2, No. 3, 1991, pp. 192-222 https://doi.org/10.1287/isre.2.3.192
  40. Mumford, E., Hirschheim, R., Fitzgerald, G., and Wood-Harper, A.T. (Eds.), Research Methods in Information Systems. Amsterdam, The Netherlands: North-Holland, 1984
  41. Newsted, P.R., Munro, M.C., and Huff, S.L., 'Data Acquisition Instruments in Management Information Systems,' In K. L. Kraemer (Ed.), The Information Systems Research Challenges: Survey Research Methods (Vol. 3, pp. 187-209), Cambridge, MA: Harvard Business School Press, 1991
  42. Nicholas, J.M. and Katz, M., 'Research Methods and Reporting Practices in Organizational Development: A Review and Some Guidelines,' Academy of Management Review, Vol. 10, No. 4, 1985, pp. 737-749 https://doi.org/10.2307/258042
  43. Nord, J.H. and Nord, G.D., 'MIS Research: Journal Status Assessment and Analysis,' Information and Management, Vol. 29, No. 1, 1995, pp. 29-42 https://doi.org/10.1016/0378-7206(95)00010-T
  44. Nunnally, J.C., Psychometric theory (2nd ed.), New York: McGraw-Hill, 1978
  45. O'Quigley, J. and Baudoin, C.E., 'Null Hypothesis and the Misuse of Statistics,' Nature, 316, 1985
  46. Orlikowski, W.J. and Baroudi, J.J., 'Studying Information Technology in Organizations: Research Approaches and Assumptions,' Information Systems Research, Vol. 2, No. 1, 1991, pp. 1-28 https://doi.org/10.1287/isre.2.1.1
  47. Pinsonneault, A. and Kraemer, K.L., 'Survey Research Methodology in Management Information Systems: An Assessment,' Journal of Management Information Systems, Vol. 10, No. 2, 1993, pp. 75-105 https://doi.org/10.1080/07421222.1993.11518001
  48. Popper, K., The Logic of Scientific Discovery. New York: Harper and Row, 1959
  49. Rosental, R. and Rosnow, R.L., Essentials of Behavioral Research: Methods and Data Analysis (2nd ed.), McGraw-Hill, 1991
  50. Rousseau, D.M., 'Issues of Level in Organizational Research: Multi-level and Cross-level Perspectives,' Research in Organizational Behavior, Vol. 7, 1987, pp. 1-37
  51. Schkade, D.A., 'Prospects for Experiments Focusing on Individuals in IS Research,' In I. Benbasat (Ed.), The Information Systems Research Challenges: Experimental Research Methods (Vol. 2, pp. 49-52), Cambridge, MA: Harvard Business School Press, 1990
  52. Schor, S. and Karten, I., 'Statistical Evaluation of Medical Journal Manuscripts,' Journal of the American Medical Association, Vol. 195, No. 13, 1966, pp. 1123-1128 https://doi.org/10.1001/jama.195.13.1123
  53. Schwab, D.P., 'Construct Validity in Organizational Behavior,' In B. W. Staw & L. L. Cummings (Eds.), Research in Organizational Behavior (Vol. 2, pp. 3-43), Greenwich, CT: JAI Press, 1980
  54. Sethi, V. and King, W., 'Construct Measurement in Information Systems Research: An Illustration in Strategic Systems,' Decision Sciences, Vol. 22, No. 3, 1991, pp. 455-472 https://doi.org/10.1111/j.1540-5915.1991.tb01274.x
  55. Straub, D.W., 'Validating Instruments in MIS Research,' MIS Quarterly, Vol. 13, No. 2, 1989, pp. 147-167 https://doi.org/10.2307/248922
  56. Tabachnick, B.G. and Fidell, L.S., Using Multivariate Statistics (2nd ed.), New York: Harper & Row, 1989
  57. Todd, P.A. and Benbasat, I., 'A Critique of Cognitive Styles Theory and Research,' In E. R. McLean (Ed.), Proceedings of First International Conference on Information Systems (pp. 82-90), Philadelphia, PA, 1980
  58. Vessey, I., 'An Investigation of the Psychological Processes Underlying the Debugging of Computer Program,' Unpublished PhD Dissertation, Department of Commerce, University of Queensland, 1984
  59. Vessey, I., Ramesh, V., and Glass, R.L, 'Research in Information Systems: An Empirical Study of Diversity in the Discipline and Its Journal,' Journal of Management Information Systems, Vol. 19, No. 2, 2002, pp. 129-174
  60. White, S.J., 'Statistical Errors in Papers in the British Journal of Psychiatry,' British Journal of Psychiatry, Vol. 135, 1979, pp. 336-342 https://doi.org/10.1192/bjp.135.4.336
  61. Wilkinson, L. and Task Force on Statistical Inference, A. B. o. S. A., 'Statistical Methods in Psychology Journals: Guidelines and Explanations,' American Psychologist, Vol. 54, No. 8, 1999, pp. 594-604 https://doi.org/10.1037/0003-066X.54.8.594
  62. Wood, G., Fundamentals of Psychological Research (3rd ed.), Boston: Little-Brown, 1981
  63. Yu, J., and Cooper, H., 'A Qualitative Review of Research Design Effects on Response Rates to Questionnaires,' Journal of Marketing Research, Vol. 36, 1983, pp. 36-44
  64. Zmud, R.W. and Boynton, A.C., 'Survey Measures and Instruments in MIS: Inventory and Appraisal,' In K. L. Kraemer (Ed.), The Information Systems Research Challenges: Survey Research Methods (Vol. 3, pp. 149-180), Cambridge, MA: Harvard Business School Press, 1991