• Asian Journal of Innovation and Policy
• /
• v.9 no.3
• /
• pp.281-307
• /
• 2020

A variety of approaches are being applied to improve the existing ex-ante evaluation by expert panels in publicly funded R&D. While the objective evaluation criteria are constantly being improved to screen and select the superior projects, alternative approaches such as random prioritization and logical modeling are also underway to overcome the conservative bias of reviewers and to secure disruptive innovation. This study intends to find critical implications for ex-ante evaluation of public R&D system from the comparison of Indonesia and Korea. For the comparative analysis, literature review and expert in-depth interviews are conducted on the national R&D system and the selection evaluation process. In Korea, the selection criteria of projects are legally promulgated for establishing an objective evaluation system, and at the program level, the major considerations in the planning process are specified by Presidential Decree. On the other hand, while Indonesia conducts R&D in 47 strategic fields largely by public research institutes (PRI) based on the non-competitive government contributions. This study draws out implications of institutionalizing the planning process at the level of program, and of increasing the ratio of contract-based competitive funding at the level of project in the national R&D portfolio.

• STI Policy Review
• /
• v.4 no.2
• /
• pp.41-54
• /
• 2013

R&D is inherently an uncertain endeavor, yet now more than ever those performing R&D with public funding are called upon to clarify the utility of their research. Calls for public accountability are mounting with the increase in constraints on government budgets due to the recent worldwide economic recession, in response to which both policymakers and researchers pay much more attention to rigorously assessing publicly funded R&D. A key issue complicating R&D evaluation in these circumstances is how to adequately account for the nature and degree of risk involved in a given R&D program or project. This study deliberates on certain issues involving the measurement of ex-ante risk in public R&D evaluation: (i) information asymmetry between R&D sponsors and performers, (ii) ambiguity in the measurement of returns in both prospective and retrospective evaluation, and (iii) the dilemma between measurement error and omitted variable bias for empirical estimation of R&D performance. The study then presents an analysis of hypothetical evaluation results that apply risk-relevant weights to the annual evaluation outcomes of South Korea's national R&D programs funded during 2006~2012. In this counterfactual re-evaluation of public R&D program performance, high-risk R&D programs turn out to receive higher evaluation than non-high-risk programs. The current study suggests that R&D evaluation ignoring ex-ante risk is not only conceptually invalid since R&D activities are intrinsically uncertain endeavors, but unfair as R&D performers are asked to be accountable for the results that were in fact out of their reach.

• Journal of Technology Innovation
• /
• v.27 no.4
• /
• pp.1-19
• /
• 2019

This study presents a new method to evaluate patents by public R&D program using patent citation network analysis. I used forward citation, degree centrality, betweenness centrality and page rank as the dependent variables which represents the quality of patents. I used primary independent variable as a dummy of public R&D program and controlled patents characteristics, applicant characteristics, technological characteristics and year effect. The empirical result shows that the patents of public R&D program is superior to other patents in regard to the number of forward citation, the degree centrality, the betweenness centrality and the page rank. This empirical result implies that patents of public R&D program directly and effectively connects technologies. Also patents from public R&D program connects important technologies.

• Journal of Korea Technology Innovation Society
• /
• v.4 no.3
• /
• pp.275-290
• /
• 2001

The frameworks for evaluation of national R&D programs reflect their various political and administrative cultures(Gibbons & Georghiou, 1987), and the structure of national R&D system. In this research the core research questions are : what is god evaluation\ulcorner What is being evaluated, by whom, by which criteria, for whom and what purpose\ulcorner In order to examine these general aims and answer these questions, in detail several objectives can be proposed on the process of this research. In this research, the national R&D programs will be considered in terms of the interface between evaluation and the wider policy-making process. The programs for case study are the Alvey program(UK), the Advanced Technology Program (US) and the Framework program(EU) . One of the characteristics of these programs is the well established evaluation systems they have. From the comparative analysis, we can withdraw some useful implication for Korean evaluation practice for national R&D programs. Evaluation system is social process and the way in which it is organized is critical to its outcome.

• Proceedings of the Technology Innovation Conference
• /
• 1999.12a
• /
• pp.216-239
• /
• 1999

Korean government has planned a large scale multidepartment-participated national R&D program to advance and improve her science and technology and the quality of life In the level of advanced(especially G-7) countries in the forthcoming 21st century. It is called as "Highly Advanced National projects" or "G7 projects", which was initiated in 1991 with 18 sub-programs to date. It has planned to be continued until 2001 with its total fund of 4, 591 billion Won, comprised of 2, 033 billion Won from the public sector and 2, 558 billion Won from the private sector. Evaluation activities, the country has carried out to date, for national R&D programs including HAN projects are focused mainly on the assessment of scientific and technological results to decide that a specific program should be continued, terminated, or modified. Thus, it is necessary for national R&D programs to be evaluated socioeconomically for the purpose of assessing the nationwide economic and social impact from the program. Socioeconomic evaluation would be told how and where the program contributed to the society, and what the socioeconomic impacts are resulted from the program. It would be useful for the means of (ⅰ) fulfillment of public accountability to legitimate the program and to reveal the expenditure of pubic fund, and (ⅱ) managemental and strategical learning to give information necessary to improve the making. program and policy decision making, The objectives of the study are to develop the methodology of modeling the socioeconomic evaluation, and build up the practical socioeconomic evaluation model of the HAN projects including scientific and technological effects. Since the HAN projects consists of 18 subprograms, it is difficult In evaluate all the subprograms simultaneously. Despite, each program is being performed under the category of HAN projects, so the common socioeconomic issues are existing, The followings are main results of the study. First, the hierarchical structure of the socioeconomic evaluation are constructed; Evaluation Perspective, Evaluation Bounds, and Evaluation Aspect. Second, based on the goals of the HAN projects, the evaluation perspectives are established as (ⅰ) the strengthening of industrial competitiveness, (ⅱ) the enhancement of national scientific and technological capability, (ⅲ) the improvement of quality of life. Third, the evaluation bounds for each evaluation objective are defined to specify the affected area. Finally, the evaluation aspects for each evaluation bounds are formulated containing essential elements describing the evaluation bounds.

• Asia-Pacific Journal of Business Venturing and Entrepreneurship
• /
• v.4 no.4
• /
• pp.1-25
• /
• 2009

The Government is putting efforts to enhance efficiency and productivity of performances of National R&D Programs by enacting 'Law on Evaluation and Management of Performances of National R&D Program'. This study is the result of development of evaluation model and indices for analysis of efficiency through meta evaluation method for the specific evaluation among evaluation systems for National R&D Program that are being carried out at government-wide level. Meta evaluation model for analysis of efficiency of evaluation system for National R&D Program designed evaluation indices by categorizing compositional factors into 5 categories, namely, evaluation context factor, evaluation resources factor, evaluation process factor, evaluation result factor and evaluation utilization factor. Furthermore, the result of verification of reliability on evaluation composition factors and evaluation indices analyzed to select evaluation items displayed Cronbach' $\alpha$ coefficient of 0.933, thereby illustrating that there is no major problem in the evaluation index system. In addition, computation of weight for each evaluation composition factor and evaluation index was analyzed by using analytical hierarchy process (AHP) technique. The result of analysis displayed overall consistency index of less than 0.1 for the investigators, thereby enabling analysis that there is consistency in investigators.

• Journal of Korea Technology Innovation Society
• /
• v.15 no.2
• /
• pp.461-479
• /
• 2012

In this study, new evaluation perspective and analysis methodology are suggested to assess the technological level of new product produced by national large R&D program and applied to case analysis of the superconducting cable technology. Logical Procedures for the assessment of the technological level include essential technology classification, analysis of evaluation factors, structure and content of assessment items, presentation method of the analysis result. Variable with technological properties are employed to analyze the relative competitiveness subjectively. The principal component analysis are applied to measure the relative competitiveness in terms of size and position. Finally the combined results of technology and right evaluation show that the technology level of DAPAS superconducting cables lies somewhere between 96.11% and 101.76% compared with a global leader in technology. The result provides sufficient evidences that it secures technological edges over other competitors. Therefore it can be called as a globally leading technology. New evaluation perspective and analysis methodology suggested can be applied usefully to assess the technology level of public large-scale R&D program.

• Journal of Environmental Health Sciences
• /
• v.48 no.2
• /
• pp.123-129
• /
• 2022

Background: The Environmental Health Action Program was a national project carried out from 2012~2021. It was aimed at developing public technologies to protect people's health from various environmental hazards. Objectives: One of the final goals of the project was "creating health benefits worth more than 179.2 billion won by reducing the environmental burden of disease." This study aims to evaluate whether the program sufficiently achieved the planned benefits. Methods: In order to secure consistency in evaluation, we applied the same equation used in the goal-setting process. It is comprised of six parameters to estimate the benefit: 1. The amount of medical expenses for environmental diseases; 2. The attributable proportion of environmental risk factors' 3. The rate of reduction in medical expenses for environmental diseases; 4. R&D project contribution; 5. The proportion of successful policy reflection; and 6. The contributions of the project. The corresponding variables were estimated at the end of the project, and the health benefits of the project were recalculated using the newly estimated variables. Results: It was estimated that a total of 195 billion won in health benefits occurred or will occur from 2015 to 2026. The main contributors for achieving the target were an increase in medical expenses for environmental diseases, a high score in the R&D project contribution, and the proportion of successful policy reflection. Conclusions: Technically, the equation used in the project is about medical expenses for environmental diseases rather than about the environmental burden of disease. There are several benefits of using the environmental burden of disease in the evaluation of public health policies. In further studies, developing a policy evaluation framework using indicators such as population attributable fraction would be needed.

• Journal of Korea Technology Innovation Society
• /
• v.16 no.1
• /
• pp.176-198
• /
• 2013

Preliminary feasibility study(PFS) was introduced in 1999 by financial authority to encourage a cautious approach to new large-scale public investment project. As it applied cost-benefit analysis as prime measure for decision-making, various issues have been arisen concerning feasibility analysis on R&D programs. This work is intended to suggest standard approaches to be established in PFS on R&D program as follows: 1. The issue questions can be induced in a standard way by 15 representative questions and their correlation with evaluation criteria. 2. The analyzing strategy can be standardized by establishing standards for classification of R&D effects and R&D logic analysis.

• Journal of Internet Computing and Services
• /
• v.24 no.6
• /
• pp.31-39
• /
• 2023

The purpose of this study is to propose a scientific performance evaluation framework for measuring and managing the overall outcome of complex types of projects that are linked to public demand-based commercialization, such as information system projects and public procurement, in integrated national R&D projects. In the case of integrated national R&D projects that involve multiple research institutes to form a single final product, and in the case of demand-based demonstration and commercialization of the project results, the existing evaluation system that evaluates performance based on the short-term outputs of the detailed tasks comprising the R&D project has limitations in evaluating the mid- and long-term effects and practicality of the integrated research products. (Moreover, as the paradigm of national R&D projects is changing to a mission-oriented one that emphasizes efficiency, there is a need to change the performance evaluation of national R&D projects to focus on the effectiveness and practicality of the results.) In this study, we propose a performance evaluation framework from a structural perspective to evaluate the completeness of each national R&D project from a practical perspective, such as its effectiveness, beyond simple short-term output, by utilizing the Hourglass model. In particular, it presents an integrated performance evaluation framework that links the top-down and bottom-up approaches leading to Tool-System-Service-Effect according to the structure of R&D projects. By applying the proposed detailed evaluation indicators and performance evaluation frame to actual national R&D projects, the validity of the indicators and the effectiveness of the proposed performance evaluation frame were verified, and these results are expected to provide academic, policy, and industrial implications for the performance evaluation system of national R&D projects that emphasize efficiency in the future.