Browse > Article

Application of Empirical Research Methods in Information Systems Research: Gaining Lessons Through Evaluation  

Kang, Shin-Cheol (한남대학교 경영정보학과)
Lee, Zoon-Ky (연세대학교 정보대학원)
Choi, Jeong-Il (Merrimack College 경영학과)
Publication Information
Asia pacific journal of information systems / v.16, no.2, 2006 , pp. 1-25 More about this Journal
Abstract
Application of appropriate research methods has assumed an important role in knowledge accumulation endeavors in the scientific research community. The current study reported here summarizes how we used the empirical methods in IS research, what we need to improve in using empirical study as research methodology through the set of comprehensive research guideline. From the survey and review of seminal and classical research guidelines, we developed our own 28 checklist for research design, statistical analysis, and conclusion, which can be commonly applied to all articles that employ inferential statistical methods in IS research. Then, we critically evaluated the usage of empirical research methods in major IS journals by using the checklist, with the goal of improving the quality of academic papers. In this study, we scrutinized four major IS journals which published empirical papers from 1991 to 2000: MIS Quarterly, Journal of MIS, Information Systems Research, and Decision Sciences. As a result of intensive evaluation work, we could highlight many areas that are lagging and call for greater attention with regard to the proper usage of empirical study in IS research. The research findings in this study can be referred as checklist and guideline when IS researcher applies the empirical method.
Keywords
Empirical Study; IS Research; Research Method; Research Design Guideline;
Citations & Related Records
연도 인용수 순위
  • Reference
1 Agarwal, R. and Prasad, J., 'The Antecedents and Consequents of User Perceptions in Information Technology Adoption,' Decision Support Systems, Vol. 22, No. 2, 1998, pp. 15-29   DOI   ScienceOn
2 Ajzen, I., 'The Theory of Planned Behavior,' Organizational Behavior and Human Decision Processes, Vol. 50, No. 2, 1991, pp. 179-211   DOI
3 Baroudi, J.J. and Olikowski, W.J., 'The Problems of Statistical Power in MIS Research,' MIS Quarterly, Vol. 13, No. 1, 1988, pp. 88-106
4 Bowers, J.W. and Courtright, J.A., Communication Research Methods, Glenview, IL, Scott, Foresman, 1984
5 Chin, W.W. and Todd, P.A., 'On the Use, Usefulness, and Ease of Use of Structural Equation Modeling in MIS research: A Note of Caution,' MIS Quarterly, Vol. 19, No. 2, 1995, pp. 237-246   DOI   ScienceOn
6 Cohen, J., 'A Coefficient of Agreement for Nominal Scales,' Educational and Psychological Measurement, Vol. 20, 1960, pp. 37-46   DOI
7 Lucas, H.C., 'Methodological Issues in Information Systems Survey Research,' In K. L. Kraemer (Ed.), The Information Systems Research Challenges: Survey Research Methods (Vol. 3, pp. 273-282). Cambridge, MA: Harvard Business School Press, 1991
8 Marx, M.H., 'Theorizing,' In M. H. Marx and F. E. Goodson (Eds.), Theories in Contemporary Psychology (2nd ed., pp. 261- 286), New York: Macmillan, 1976b
9 Moore, G.C. and Benbasat, I., 'Development of an Instrument to Measure the Perceptions of Adopting an Information Technology Innovation,' Information Systems Research, Vol. 2, No. 3, 1991, pp. 192-222   DOI
10 Gaither, N. and Glorfeld, L., 'An Evaluation of the Use of Tests of Significance in Organizational Behavior Research,' Academy of Management Review, Vol. 10, No. 4, 1985, pp. 787-793   DOI
11 Pinsonneault, A. and Kraemer, K.L., 'Survey Research Methodology in Management Information Systems: An Assessment,' Journal of Management Information Systems, Vol. 10, No. 2, 1993, pp. 75-105   DOI
12 Rousseau, D.M., 'Issues of Level in Organizational Research: Multi-level and Cross-level Perspectives,' Research in Organizational Behavior, Vol. 7, 1987, pp. 1-37
13 Wood, G., Fundamentals of Psychological Research (3rd ed.), Boston: Little-Brown, 1981
14 Hamilton, S. and Ives, B., 'MIS Research Strategies,' Information and Management, Vol. 5, No. 4, 1982, pp. 339-347   DOI   ScienceOn
15 Newsted, P.R., Munro, M.C., and Huff, S.L., 'Data Acquisition Instruments in Management Information Systems,' In K. L. Kraemer (Ed.), The Information Systems Research Challenges: Survey Research Methods (Vol. 3, pp. 187-209), Cambridge, MA: Harvard Business School Press, 1991
16 Schwab, D.P., 'Construct Validity in Organizational Behavior,' In B. W. Staw & L. L. Cummings (Eds.), Research in Organizational Behavior (Vol. 2, pp. 3-43), Greenwich, CT: JAI Press, 1980
17 Cohen, J., Statistical Power Analysis for the Behavioral Sciences (2nd ed.), Hillsdale, NJ: Erlbaum, 1988
18 Jarvenpaa, S., 'The Effect of Task Demands and Graphical Format on Information Processing Strategies,' Management Science, Vol. 35, No. 3, 1989, pp. 285-303   DOI   ScienceOn
19 Bailar, J.C., III., and Mosteller, F., 'Guidelines for Statistical Reporting in Articles for Medical Journals,' Annals of Internal Medicine, Vol. 108, 1988, pp. 266-273   DOI   ScienceOn
20 Popper, K., The Logic of Scientific Discovery. New York: Harper and Row, 1959
21 Schor, S. and Karten, I., 'Statistical Evaluation of Medical Journal Manuscripts,' Journal of the American Medical Association, Vol. 195, No. 13, 1966, pp. 1123-1128   DOI
22 O'Quigley, J. and Baudoin, C.E., 'Null Hypothesis and the Misuse of Statistics,' Nature, 316, 1985
23 Schkade, D.A., 'Prospects for Experiments Focusing on Individuals in IS Research,' In I. Benbasat (Ed.), The Information Systems Research Challenges: Experimental Research Methods (Vol. 2, pp. 49-52), Cambridge, MA: Harvard Business School Press, 1990
24 Orlikowski, W.J. and Baroudi, J.J., 'Studying Information Technology in Organizations: Research Approaches and Assumptions,' Information Systems Research, Vol. 2, No. 1, 1991, pp. 1-28   DOI
25 Jarvenpaa, S., Dickson, G., and DeSanctis, G., 'Methodological Issues in Experimental IS Research: Experiences and Recommendations,' MIS Quarterly, Vol. 9, No. 2, 1985, pp. 237-246
26 Marx, M.H., 'Formal Theory,' In M. H. Marx and F. E. Goodson (Eds.), Theories in Contemporary Psychology (2nd ed., pp. 234-260), New York: Macmillan, 1976a
27 Todd, P.A. and Benbasat, I., 'A Critique of Cognitive Styles Theory and Research,' In E. R. McLean (Ed.), Proceedings of First International Conference on Information Systems (pp. 82-90), Philadelphia, PA, 1980
28 Bacharach, S.B., 'Organizational Theories: Some Criteria for Evaluation,' Academy of Management Review, Vol. 14, No. 4, 1989, pp. 496-515   DOI
29 Hair, J.F., Anderson, R.E., Tatham, R.L., and Black, W.C., Multivariate Data Analysis (5th ed.), Upper Saddle River, NJ: Prentice Hall, 1998
30 Rosental, R. and Rosnow, R.L., Essentials of Behavioral Research: Methods and Data Analysis (2nd ed.), McGraw-Hill, 1991
31 Sethi, V. and King, W., 'Construct Measurement in Information Systems Research: An Illustration in Strategic Systems,' Decision Sciences, Vol. 22, No. 3, 1991, pp. 455-472   DOI
32 Davis, F.D., 'Perceived Usefulness, Perceived Ease of Use and User Acceptance of Information Technology,' MIS Quarterly, Vol. 13, No. 3, 1989, pp. 319-339   DOI   ScienceOn
33 Alavi, M. and Carlson, P., 'A Review of MIS Research and Disciplinary Development,' Journal of Management Information Systems, Vol. 8, No. 4, 1992, pp. 45-62   DOI
34 Nicholas, J.M. and Katz, M., 'Research Methods and Reporting Practices in Organizational Development: A Review and Some Guidelines,' Academy of Management Review, Vol. 10, No. 4, 1985, pp. 737-749   DOI
35 Vessey, I., 'An Investigation of the Psychological Processes Underlying the Debugging of Computer Program,' Unpublished PhD Dissertation, Department of Commerce, University of Queensland, 1984
36 Adams, D.A., Nelson, R.R., and Todd, P.A., 'Perceived Usefulness, Ease of Use, and Usage of Information Technology: A Replication,' MIS Quarterly, Vol. 16, No. 2, 1992, pp. 227-247   DOI   ScienceOn
37 Boudreau, M., Gefen, D., and Straub, D. W., 'Validation in IS Research: A State- of-the Art Assessment,' MIS Quarterly, Vol. 25, No. 1, 2001, pp. 1-16   DOI   ScienceOn
38 Straub, D.W., 'Validating Instruments in MIS Research,' MIS Quarterly, Vol. 13, No. 2, 1989, pp. 147-167   DOI   ScienceOn
39 Campion, M.A., 'Article Review Checklist: A Criterion Checklist for Reviewing Research Articles in Applied Psychology,' Personnel Psychology, Vol. 46, 1993, pp. 705-718   DOI   ScienceOn
40 Chau, P.Y.K., 'On the Use of Construct Reliability in MIS Research: A Meta-Analysis,' Information and Management, Vol. 35, No. 4, 1999, pp. 217-227   DOI   ScienceOn
41 Vessey, I., Ramesh, V., and Glass, R.L, 'Research in Information Systems: An Empirical Study of Diversity in the Discipline and Its Journal,' Journal of Management Information Systems, Vol. 19, No. 2, 2002, pp. 129-174
42 Miles, M.B. and Huberman, A.M., Qualitative Data Analysis: An Expanded Sourcebook, Thousand Oaks, CA: Sage Publications, Inc, 1994
43 Hughes, C.T. and Gibson, M.L., 'Students as Surrogates for Managers in a Decision- making Environment: An Experimental Study,' Journal of Management Information Systems, Vol. 8, No. 2, 1991, pp. 153-166   DOI
44 Kerlinger, F.N., Foundations of Behavioral Research (3rd ed.), New York: CBS College Publishing, 1986
45 Lee, A.S., 'A Scientific Methodology for MIS Case Studies,' MIS Quarterly, Vol. 13, No. 1, 1989, pp. 33-49   DOI   ScienceOn
46 Nord, J.H. and Nord, G.D., 'MIS Research: Journal Status Assessment and Analysis,' Information and Management, Vol. 29, No. 1, 1995, pp. 29-42   DOI   ScienceOn
47 Tabachnick, B.G. and Fidell, L.S., Using Multivariate Statistics (2nd ed.), New York: Harper & Row, 1989
48 Yu, J., and Cooper, H., 'A Qualitative Review of Research Design Effects on Response Rates to Questionnaires,' Journal of Marketing Research, Vol. 36, 1983, pp. 36-44
49 Landis, J. and Koch, G.G., 'The Measurement of Observer Agreement for Categorical Data,' Biometrics, Vol. 33, 1977, pp. 159-174   DOI   ScienceOn
50 Mumford, E., Hirschheim, R., Fitzgerald, G., and Wood-Harper, A.T. (Eds.), Research Methods in Information Systems. Amsterdam, The Netherlands: North-Holland, 1984
51 Zmud, R.W. and Boynton, A.C., 'Survey Measures and Instruments in MIS: Inventory and Appraisal,' In K. L. Kraemer (Ed.), The Information Systems Research Challenges: Survey Research Methods (Vol. 3, pp. 149-180), Cambridge, MA: Harvard Business School Press, 1991
52 Culnan, M. and Swanson, E., 'Research in Management Information Systems, 1980- 1984; Points of Work and Reference,' MIS Quarterly, Vol. 7, No. 3, 1986, pp. 286-303
53 Wilkinson, L. and Task Force on Statistical Inference, A. B. o. S. A., 'Statistical Methods in Psychology Journals: Guidelines and Explanations,' American Psychologist, Vol. 54, No. 8, 1999, pp. 594-604   DOI
54 Delucchi, K.L., 'The Use and Misuse of Chi-Square: Lewis and Burke Revisited,' Psychological Bulletin, Vol. 94, No. 1, pp. 1987, 166-176   DOI
55 Gardner, W., On the Reliability of Sequential Data: Measurement, Meaning and Correction, Mahwah, NJ: Erlbaum, 1995
56 White, S.J., 'Statistical Errors in Papers in the British Journal of Psychiatry,' British Journal of Psychiatry, Vol. 135, 1979, pp. 336-342   DOI
57 Burnett, J.J. and Dunne, P.M., 'An Appraisal of the Use of Student Subjects in Marketing Research,' Journal of Business Research, Vol. 14, No. 4, 1988, pp. 329-343   DOI   ScienceOn
58 Meyers, L.S. and Grossen, N.E., Behavioral Research: Theory, Procedure, and Design, San Francisco, CA: W.H. Freeman and Company, 1974
59 Mitchell, T.R., 'An Evaluation of the Validity of Correlational Research Conducted in Organizations,' Academy of Management Review, Vol. 10, No. 2, 1985, pp. 192-205   DOI
60 LaTour, S.A. and Miniard, P.W., 'The Misuse of Repeated Measures Analysis in Marketing Research,' Journal of Mar Marketing Research,' Journal of Marketing Research, Vol. 20, No. 1, 1983, pp. 45-47   DOI   ScienceOn
61 Keen, P., MIS Research: Reference Disciplines and a Cumulative Tradition, Paper presented at the The First International Conference on Information Systems, Philadelphia, PA, 1980
62 Kraemer, K.L. and Dutton, W.H., 'Survey Research in the Study of Management Information Systems,' In K. L. Kraemer (Ed.), The Information Systems Research Challenges: Survey Research Methods (Vol. 3, pp. 3-57), Cambridge, MA: Harvard Business School Press, 1991
63 Nunnally, J.C., Psychometric theory (2nd ed.), New York: McGraw-Hill, 1978
64 Ajzen, I., 'From Intensions to Actions: A Theory of Planned Behavior. In J. Kuhl & J. Beckmann (Eds.),' Action Control: From Cognition to Behavior (pp. 11-39). New York: Springer, 1985