Browse > Article

A Study on the Meta Evaluation for Defense R&D Programs  

Kim, Soon-Yeong (Dept. of Technology Evaluation, Defense Agency for Technology and Quality)
Ha, Kyu-Soo (The Graduate School of Venture, Hoseo University)
Publication Information
Journal of Korean Society of Industrial and Systems Engineering / v.32, no.3, 2009 , pp. 59-70 More about this Journal
Abstract
This study is the result of meta evaluation for the self evaluation of defense R&D programs in Korea by using meta evaluating indicators. The overall meta evaluation result of defense R&D programs gained 74.3 points out of 100, so it was evaluated as 'Good'. But it demonstrated that further improvement for overall system of defense R&D programs evaluation is required. And especially, it demonstrated that more alternatives are necessary in order to improve the utilizations and the feedbacks of evaluation results. The evaluation context component gained 80.2 points out of 100, so it was evaluated as 'Very Good'. The evaluation input component gained 73.1 points out of 100, so it was evaluated as 'Good'. The evaluation process component gained 74.8 points out of 100, so it was evaluated as 'Good'. And the evaluation outcome component gained 69.0 points out of 100, so it was evaluated as 'Good'. Basic model of meta evaluation was derived from the literature review and brain storming. And this meta evaluation model was determined by adopting the result of experts who performed evaluations for defense R&D programs in recent years. The reliability of components and items was verified by Cronbach's a coefficient. It was over 0.6 in evaluation components and items. And the reliability of evaluation context was 0.877, that of evaluation input was 0.755, that of evaluation process was 0.755, that of evaluation output was 0.755 respectively. From the analysis, it is attempted to identify possible problems and to find out the ways of improvements related to the self evaluation system of defense R&D programs. The ultimate objective of this study is to manage the programs effectively and improve the reliability and the objectiveness of the defense R&D programs.
Keywords
Defense R&D Programs; Performance Evaluation; Self Evaluation; Meta evaluation;
Citations & Related Records
연도 인용수 순위
  • Reference
1 American Evaluation Association "Guidelines for Evaluator," by Shadish W. R., D. L. Newman, M. A. Scheirer and C.; Wye, San francisco, Jossey-Bass, 1995
2 Chelimsky, Eleanor; "Program Evaluation: Patterns and Directions," 1-35, Washington DC, The American Society for Public Administration, 1985
3 Hong H. D.; "Meta-evaluation of National Large-Scale R&D Programs : A Comparison of Evaluation Systems of 6 National R&D Programs," University of Manchester, 2000
4 OECD, "Improving Evaluation Practices-Best Practice Guidelines for Evaluation and Background Paper," Paris, 1999
5 Stufflebeam L.; "Metaevaluation : Concepts, Standard and Uses," Educational Evaluation Methodology, The JHU Press, 1981
6 과학기술혁신본부; “2008년도 국가연구개발사업 자체평가매뉴얼”, 2008
7 Cook T D. and Gruder C. L.; "Metaevaluation Research," Evaluation Quarterly, 2(1) : 5-55, 1978   DOI
8 노유진, “우리나라 정부부처에 있어서 정책평가결과의 저활용요인에 관한 연구”, 행정논총, 43(4) : 245-276, 2005
9 박종수; “정보화 사업 메타평가를 위한 CIPOU 모형개발”, 박사학위논문, 충남대학교 대학원, 2003
10 김병철, “정부출연연구기관 평가에 대한 메타평가 모형설계 및 적용”, 박사학위논문, 서울시립대학교 대학원, 2008
11 Worthen, Blaine R., James R. Sanders and Jody L.; Fitzpatrisk, "Program Evaluation: Alternative Approaches and Practical Guidelines," Second Edition, White Plains, Longman Publishers, 1997
12 이형우; “국가연구개발사업 평가체계에 관한 소고”, 정책분석평가학회보, 15(2) : 191-219, 2005
13 Joint Committee on Standards for Educational Evaluation; "The Programme Evaluation Standards," Thousand Oaks, Sage Publications, 1994
14 이찬구; “영국의 범부처간 과학기술정책 평가제도-메타평가와 한국에의 시사점”, 한국행정논문집, 15(4) : 809-833, 2003
15 과학기술부; “국가연구개발사업 등의 성과평가 및 성과관리에 관한 법률”, 법률 제7808호, 2005
16 Scriven M.; "Evaluation Thesaurus(4th ed.)," Newbury Park, CA, Sage Publications, 1991
17 Larson, Richard C and Leni Berliner; "On Evaluating Valuations," Policy Science, 16(2) : 147-163, 1983   DOI
18 Stevenson, John F., Richard H. Longabaugh and Dwight N. Mcneil; "Metaevaluation in Human Services," The Evaluator and Management, Beverly Hills, CA, Sage Publications, 2000
19 류영수; “기술영향평가의 메타평가 모형개발 및 적용”, 박사학위논문, 한양대학교 대학원, 2007
20 이학식, 임지훈; “SPSS 14.0 매뉴얼”, 법문사, 2008
21 방위사업청; “방위력개선사업관리규정”, 2006
22 Leviton, Laura C and Edward F. X. Hughes; "Research on the Utilization of Evaluations: A Review and Synthesis," Evaluation Review, 5(4) : 1981   DOI
23 Yi, Chan Goo, "The National R&D Evaluation System in the UK : Meta-evaluation and Applicability to the Korea Case," Thesis for the degree of PhD, PREST, University of Manchester, 2003