기술혁신학회지 (Journal of Korea Technology Innovation Society)

  • 제17권4호
  • /
  • Pages.703-732
  • /
  • 2014
  • /
  • 1598-2912(pISSN)
  • /
  • 2713-8666(eISSN)
한국기술혁신학회 (Korea Technology Innovation Society)
권호 표지

국가연구개발사업 메타평가모형 설계와 응용

The Design and Application of a Meta-evaluation Model for National R&D Programs

  • 투고 : 2014.09.08
  • 심사 : 2014.12.01
  • 발행 : 2014.12.01
PDF 다운로드
⟨ 이전 논문 다음 논문 ⟩

초록

국가연구개발사업에 대한 메타평가는 과학기술수요와 목표와 연계된 국가전체의 연구개발시스템의 틀 안에서 구축되어야 한다. 연구개발사업평가시스템은 투입, 수행, 산출, 활용의 전 과정으로 이해될 수 있다. 전문가 24명이 참여한 전문가 델파이기법을 통하여 24개의 평가지표가 개발되었다. 메타평가 모델은 중요한 관점 8개의 하위시스템을 포함하고 있다. 8개 하위시스템에는 기획의 적실성, 정보의 충실성, 평가투입에서 평가자의 적정성, 평가수행과정과 방법의 적절성, 평가활용보고와 응용의 유용성 등이 포함된다. 메타평가모형의 활용방법으로서 평정점수의 평균은 5점척도 기준으로 2.73이며, 이는 평가점수의 중앙값 3.0보다 낮게 나타났다. 평가결과의 환류는 연구개발프로그램의 개선에 특별히 강화될 것을 제언한다.

Meta-evaluation for national R&D programs should be established within the framework of whole R&D system linked with science and technology needs and objectives. The R&D program evaluation system can be understood as a circulation of input, implementation, output, and utilization. In the result of the Delphi survey, which consisted of twenty-four experts (performed in three rounds), twenty-four indices were developed. The meta-evaluation model included sub-items which were important points of view of it as follow (eight items): propriety of planning, sufficiency of information, and propriety of evaluators in evaluation input; appropriateness of method and appropriateness of procedure in evaluation implementation; credibility of output in evaluation output; and usefulness of report and application of evaluation utilization. As the application of the meta-evaluation model, the total mean was lower than 3.0 of the median value of 2.73 (5 point Likert scale). Finally, it was suggested that the feedback of evaluation results should be more enforced to R&D program improvement particularly (evaluation utilization 2.50).

키워드

참고문헌

  1. American Evaluation Association (1995), Guiding Principles for Evaluators, by Shadish, W. R. D. L. Newman, M. A. Scheirer and C. Wye, San francisco: Jossey-Bass.
  2. Bush, V. (1960), Science: The Endless Frontier, A Report to the President on a Program for Postwar Scientific Research by Vannevar Bush, Washington: National Science Foundation.
  3. Chang, I. B. (1995), "An Analysis of Factors Influencing the Government-Sponsored R&D Performance", Thesis for the degree of PhD, Sungkyunkwan University.
  4. Chen, H. (2005), Practical Program Evaluation: Assessing and Improving Planning, Implementation and Effectiveness, Thousand Oaks and London and New Delhi: Sage Publications.
  5. Cho, W. H., Woo, J. P. and Song, C. H. (2010), "Practical Analysis of Developed Bibliometric Indicators of Evaluation of National R&D Programs: Case Study of World Class University Program", Journal of Korea Technology Innovation Society, 13(3): 494-512.
  6. Cook, T. D. and Gruder, C. L. (1978), "Meta-evaluation Research", Evaluation Quarterly, 2(1): 5-55. https://doi.org/10.1177/0193841X7800200101
  7. Dror, Y. (1971), Ventures in Policy Sciences, New York: American Elsevier Publishing.
  8. European Commission (2004), Evaluation EU Activities: A Practical Guide for the Commission Service, Luxembourg: DG Budget-Evaluation Unit, July 2004.
  9. Geisler, E. (1994), "Key Output Indicators in Performance Evaluation of Research and Development Organization", Technological Forecasting and Social Change, 42: 189-203.
  10. Heo, J. G., Kim, H. D., Cho, Y. D., Cho, S. M. and Cho, S. R. (2008), "Developing Bibliometric Indicators for Analysis and Evaluation of National R&D Programs", Journal of Korea Technology Innovation Society, 11(3): 376-399.
  11. Hong, H. D. (2000), "Meta-evaluation of National Large-Scale R&D Programs: A Comparison of Evaluation Systems of 6 National R&D Programs", Thesis for the degree of PhD, Policy Research in Engineering Science and Technology (PREST), University of Manchester.
  12. Hong, H. D. and Mark, B. (2003), R&D Program Evaluation: Theory and Practice, England: Ashgate Publishing Ltd.
  13. Hong, S. G. (2007), "A Study on Meta-evaluation for National R&D Evaluation System", Kookmin University.
  14. Jang., D. H. and Kang, G. M. (2014), "A Study on the Improvement of a Patent Evaluation Indicator in Evaluating Government R&D Performance: Focusing on a Pilot Test of the Ministry of Maritime Affairs and Fisheries' Support Program", Korean Policy Studies Review, 23(2): 65-91.
  15. Joint Committee on Standards for Educational Evaluation(1994), The Program Evaluation Standards: How to Assess Evaluations of Educational Program, Second Edition, Thousands Oaks and New Delhi and London: Sage Publications.
  16. Kim, J. B. (1993), Science and Technology Policy Theory, Daeyoung Publications.
  17. Kim, M. S. (2003), Theory of Public Policy Evaluation, Pakyoung Publications.
  18. Kim, S. Y. (2009), "A Study on the Development of Meta-evaluation Indicators for Defense R&D Programs by Using FA/AHP Methods", Korea Technology Innovation Society, 12(1): 113-136.
  19. Larson, R. and Berliner, L. (1983), "On Evaluating Evaluations", Policy Sciences, 16(2): 147-163. https://doi.org/10.1007/BF00138348
  20. Lee, J., Kim, J. S. and Kook, Y. Y. (2010), "A Study on the Process Improvement for Interoperability between National R&D Performance Evaluation and Budget Planning", Journal of Korea Technology Innovation Society, 13(1): 44-67.
  21. MSIP and KISTEP (2014a), Analysis of National R&D Programs, Annual Report.
  22. MSIP and KISTEP (2014b), Evaluation Results of National R&D Programs, Annual Report.
  23. MSIP and KISTEP (2014c), Guideline for Self-evaluation on National R&D Programs.
  24. OECD (1998), Best Practice Guidelines for Evaluation, Puma Policy Brief, 5(May), Public Management Service.
  25. OECD (2002), Frascati Manual: Standard Practice for Surveys of Research and Experimental Development, Paris: OECD.
  26. Owen, J. M. and Rogers, P. J. (1999), Program Evaluation: Forms and Approaches, International edition, London and Thousand Oaks and New Delhi: Sage Publications.
  27. Park, J. S. (2003), "Design and Application of the Meta-evaluation for the National IT R&D Project", Thesis for the degree of PhD, Chungnam National University.
  28. Rho, H. J. (2006), Policy Evaluation Theory, Bobmun Publications.
  29. Rossi, P. H., Lipsey, M. W. and Freeman, H. E. (2004), Evaluation: A Systematic Approach, Seventh Edition, Thousand Oaks and London and New Delhi: Sage Publications.
  30. Ruegg, R. and Feller, I. (2003), A Toolkit for Evaluating Public R&D Investment: Models, Methods and Findings from ATP's First Decade, Maryland: Economic Assessment Office, NIST.
  31. Ryu, Y. S. (2009), "A Study on Meta-evaluation of National R&D Evaluation System in Korea", Korean Policy Studies Review, 18(1): 158-185.
  32. Scriven, M. (1991), Evaluation Thesaurus, Fourth Edition, Newbury Park London New Delhi, Newbury Park and London and New Delhi: Sage Publications.
  33. Scriven, M. (2006), "Key Evaluation Checklist(KEC), The Evaluation Center, Joint Committee", http://www.wmich.edu/evalctr/checklist s/kec_june06.pdf. (2014. 6.).
  34. Shin, Y. (2008), "Relations between the Evaluation Results and Budgetary Allocations of the National R&D Programs", Korean Journal of Policy Analysis and Evaluation, 18(2): 151-173.
  35. Shulman, S. (2002), Trouble on The Endless Frontier: Science, Invention and the Erosion of the Technological Commons, Washington, DC: New America Foundation and Public Knowledge.
  36. Stufflebeam, D. L. (1974), Meta Evaluation, Occasional Paper Series, 3, Kalamazoo: The Evaluation Center, Western Michigan University.
  37. Stufflebeam, D. L. (1981), "Meta Evaluation: Concept, Standard, and Uses", In Berk, R. A. (ed.), Educational Evaluation Methodology: The State of the Art, 146-163, Maryland: The Johns Hopkins University Press.
  38. Vedung, E. (1997), Public Policy and Program Evaluation, New Brunswick and London: Transaction Publishers.
  39. Yang, H. S. (2003), "A Study on Methodology of National R&D Program Evaluation and Reformation of Project Based System", Sejong University.
  40. Yi, C. O. (1997), "Meta-evaluation for National R&D: IT Project", Thesis for the degree of PhD, Chungnam National University.
  41. Yi, C. O. (2003), "The National R&D Evaluation System in the UK: Meta-evaluation and Applicability to the Korea Case", Thesis for the degree of PhD, Policy Research in Engineering Science and Technology(PREST), University of Manchester.