Browse > Article

The Design and Application of a Meta-evaluation Model for National R&D Programs  

Ryu, Youngsoo (한국과학기술기획평가원)
Choi, Byungdae (한양대학교 행정학과)
Choi, Sang Ok (고려대학교 행정학과)
Publication Information
Journal of Korea Technology Innovation Society / v.17, no.4, 2014 , pp. 703-732 More about this Journal
Abstract
Meta-evaluation for national R&D programs should be established within the framework of whole R&D system linked with science and technology needs and objectives. The R&D program evaluation system can be understood as a circulation of input, implementation, output, and utilization. In the result of the Delphi survey, which consisted of twenty-four experts (performed in three rounds), twenty-four indices were developed. The meta-evaluation model included sub-items which were important points of view of it as follow (eight items): propriety of planning, sufficiency of information, and propriety of evaluators in evaluation input; appropriateness of method and appropriateness of procedure in evaluation implementation; credibility of output in evaluation output; and usefulness of report and application of evaluation utilization. As the application of the meta-evaluation model, the total mean was lower than 3.0 of the median value of 2.73 (5 point Likert scale). Finally, it was suggested that the feedback of evaluation results should be more enforced to R&D program improvement particularly (evaluation utilization 2.50).
Keywords
R&D Program Evaluation; Meta-evaluation; Evaluation Indicators; Government capacity;
Citations & Related Records
연도 인용수 순위
  • Reference
1 American Evaluation Association (1995), Guiding Principles for Evaluators, by Shadish, W. R. D. L. Newman, M. A. Scheirer and C. Wye, San francisco: Jossey-Bass.
2 Bush, V. (1960), Science: The Endless Frontier, A Report to the President on a Program for Postwar Scientific Research by Vannevar Bush, Washington: National Science Foundation.
3 Chang, I. B. (1995), "An Analysis of Factors Influencing the Government-Sponsored R&D Performance", Thesis for the degree of PhD, Sungkyunkwan University.
4 Chen, H. (2005), Practical Program Evaluation: Assessing and Improving Planning, Implementation and Effectiveness, Thousand Oaks and London and New Delhi: Sage Publications.
5 Cho, W. H., Woo, J. P. and Song, C. H. (2010), "Practical Analysis of Developed Bibliometric Indicators of Evaluation of National R&D Programs: Case Study of World Class University Program", Journal of Korea Technology Innovation Society, 13(3): 494-512.
6 Cook, T. D. and Gruder, C. L. (1978), "Meta-evaluation Research", Evaluation Quarterly, 2(1): 5-55.   DOI
7 Dror, Y. (1971), Ventures in Policy Sciences, New York: American Elsevier Publishing.
8 European Commission (2004), Evaluation EU Activities: A Practical Guide for the Commission Service, Luxembourg: DG Budget-Evaluation Unit, July 2004.
9 Geisler, E. (1994), "Key Output Indicators in Performance Evaluation of Research and Development Organization", Technological Forecasting and Social Change, 42: 189-203.
10 Heo, J. G., Kim, H. D., Cho, Y. D., Cho, S. M. and Cho, S. R. (2008), "Developing Bibliometric Indicators for Analysis and Evaluation of National R&D Programs", Journal of Korea Technology Innovation Society, 11(3): 376-399.
11 Hong, H. D. (2000), "Meta-evaluation of National Large-Scale R&D Programs: A Comparison of Evaluation Systems of 6 National R&D Programs", Thesis for the degree of PhD, Policy Research in Engineering Science and Technology (PREST), University of Manchester.
12 Hong, H. D. and Mark, B. (2003), R&D Program Evaluation: Theory and Practice, England: Ashgate Publishing Ltd.
13 Hong, S. G. (2007), "A Study on Meta-evaluation for National R&D Evaluation System", Kookmin University.
14 Jang., D. H. and Kang, G. M. (2014), "A Study on the Improvement of a Patent Evaluation Indicator in Evaluating Government R&D Performance: Focusing on a Pilot Test of the Ministry of Maritime Affairs and Fisheries' Support Program", Korean Policy Studies Review, 23(2): 65-91.
15 Joint Committee on Standards for Educational Evaluation(1994), The Program Evaluation Standards: How to Assess Evaluations of Educational Program, Second Edition, Thousands Oaks and New Delhi and London: Sage Publications.
16 Kim, J. B. (1993), Science and Technology Policy Theory, Daeyoung Publications.
17 Lee, J., Kim, J. S. and Kook, Y. Y. (2010), "A Study on the Process Improvement for Interoperability between National R&D Performance Evaluation and Budget Planning", Journal of Korea Technology Innovation Society, 13(1): 44-67.
18 Kim, M. S. (2003), Theory of Public Policy Evaluation, Pakyoung Publications.
19 Kim, S. Y. (2009), "A Study on the Development of Meta-evaluation Indicators for Defense R&D Programs by Using FA/AHP Methods", Korea Technology Innovation Society, 12(1): 113-136.
20 Larson, R. and Berliner, L. (1983), "On Evaluating Evaluations", Policy Sciences, 16(2): 147-163.   DOI   ScienceOn
21 MSIP and KISTEP (2014a), Analysis of National R&D Programs, Annual Report.
22 MSIP and KISTEP (2014b), Evaluation Results of National R&D Programs, Annual Report.
23 MSIP and KISTEP (2014c), Guideline for Self-evaluation on National R&D Programs.
24 OECD (1998), Best Practice Guidelines for Evaluation, Puma Policy Brief, 5(May), Public Management Service.
25 OECD (2002), Frascati Manual: Standard Practice for Surveys of Research and Experimental Development, Paris: OECD.
26 Owen, J. M. and Rogers, P. J. (1999), Program Evaluation: Forms and Approaches, International edition, London and Thousand Oaks and New Delhi: Sage Publications.
27 Park, J. S. (2003), "Design and Application of the Meta-evaluation for the National IT R&D Project", Thesis for the degree of PhD, Chungnam National University.
28 Rho, H. J. (2006), Policy Evaluation Theory, Bobmun Publications.
29 Rossi, P. H., Lipsey, M. W. and Freeman, H. E. (2004), Evaluation: A Systematic Approach, Seventh Edition, Thousand Oaks and London and New Delhi: Sage Publications.
30 Ruegg, R. and Feller, I. (2003), A Toolkit for Evaluating Public R&D Investment: Models, Methods and Findings from ATP's First Decade, Maryland: Economic Assessment Office, NIST.
31 Shin, Y. (2008), "Relations between the Evaluation Results and Budgetary Allocations of the National R&D Programs", Korean Journal of Policy Analysis and Evaluation, 18(2): 151-173.
32 Ryu, Y. S. (2009), "A Study on Meta-evaluation of National R&D Evaluation System in Korea", Korean Policy Studies Review, 18(1): 158-185.
33 Scriven, M. (1991), Evaluation Thesaurus, Fourth Edition, Newbury Park London New Delhi, Newbury Park and London and New Delhi: Sage Publications.
34 Scriven, M. (2006), "Key Evaluation Checklist(KEC), The Evaluation Center, Joint Committee", http://www.wmich.edu/evalctr/checklist s/kec_june06.pdf. (2014. 6.).
35 Shulman, S. (2002), Trouble on The Endless Frontier: Science, Invention and the Erosion of the Technological Commons, Washington, DC: New America Foundation and Public Knowledge.
36 Stufflebeam, D. L. (1974), Meta Evaluation, Occasional Paper Series, 3, Kalamazoo: The Evaluation Center, Western Michigan University.
37 Stufflebeam, D. L. (1981), "Meta Evaluation: Concept, Standard, and Uses", In Berk, R. A. (ed.), Educational Evaluation Methodology: The State of the Art, 146-163, Maryland: The Johns Hopkins University Press.
38 Vedung, E. (1997), Public Policy and Program Evaluation, New Brunswick and London: Transaction Publishers.
39 Yang, H. S. (2003), "A Study on Methodology of National R&D Program Evaluation and Reformation of Project Based System", Sejong University.
40 Yi, C. O. (1997), "Meta-evaluation for National R&D: IT Project", Thesis for the degree of PhD, Chungnam National University.
41 Yi, C. O. (2003), "The National R&D Evaluation System in the UK: Meta-evaluation and Applicability to the Korea Case", Thesis for the degree of PhD, Policy Research in Engineering Science and Technology(PREST), University of Manchester.