Determining criteria and weights for evaluating the socio-economic efficiency of scientific and technological topics/projects after acceptance

Article history: Received October 29 2019 Received in revised format November 21 2019 Accepted November 29 2019 Available online December 2 2019 Evaluating the socio-economic efficiency of scientific and technological topics/projects after acceptance is very difficult because it is not easy to find common ground on the criteria for evaluating the socio-economic efficiency of topics/ projects of different research fields. Hence, depending on researchers’ views, the evaluation criteria or the weight of evaluating the socio-economic efficiency of each topic/project should be different. This study uses the expert method and the Analytic Hierarchy Process (AHP) method to determine the criteria and to calculate weights for criteria on five scientific fields according to scientific sub-disciplines in Binh Dinh province, including agricultural science, natural science, social sciences and humanities, medical science, engineering and technological sciences. Our research shows that the evaluation of the socio-economic efficiency of the provincial topics/projects should be based on eight aspects, including science, technology, economics, environment, culture society, information management, and education. At the same time, the results also identified a set of common criteria for evaluating the socio-economic efficiency of topics/projects and suggested how to calculate points and to rank the effectiveness of topics/projects. However, the weight of the criteria and the composition of each criterion is different. by the authors; licensee Growing Science, Canada 20 © 20


Introduction
Scientific research and technological development are the incentives for development, so the countries' governments have invested capital and human resources for these activities. Besides carrying out the research, evaluating socio-economic efficiency of science and technology topics/projects after acceptance has also been interested in by scientists. However, evaluating socio-economic efficiency of science and technology topics/projects after acceptance is very difficult because of difficuties in finding common ground on the criteria for evaluating the socio-economic efficiency of different research fields topics/ projects. Therefore, depending on the opinion of scientists, the criteria or the weight of evaluating the socio-economic efficiency of each topic/project are different. To do this study, we conducted in-depth interviews with 50 experts from various research fields according to scientific sub-disciplines in Binh Dinh province such as agricultural science, natural science, social sciences and humanities, medical science, engineering and technological sciences to determine evaluation criteria and composition of criteria for scientific fields. Next, we use the Analytical hierarchy process (AHP) method (Saaty, 1980) and discuss with the experts to determine the evaluation weights for each criterion for each scientific field as well as the component weights for each criterion. The rest of the paper is structured as follows: section 2 is literature review, followed by the research 108 methodology in section 3, results and discussion are given in section 4, and finally conclusions and some implications are presented in section 5.

Literature review
In the world, there are many studies on evaluating the socio-economic efficiency of topics/projects. However, depending on the evaluation perspective and the evaluation field, the studies have different evaluation criteria and evaluation perspective as the following studies: Airaghi et al. (1999) investigated the contributions of social sciences and humanities in Europe, the results suggested that the contributions of topics were expressed through the aspects of science and technology, society and politics. In which, the impact on science and technology is a change in research that contributes to break previous research models and influences on future research, such as creating the capacity for establishing new perspectives and ideas and affecting future specialized studies, impact on society such as human rights issues, social cohesion, economic cohesion, employment, public health, culture, consumer benefits, etc. and the impact on politics is the application of research results as a basis for making policies and laws. Kolsch et al. (2008) introduced a measure of social impact by SeeBalance method to evaluate the social effectiveness of topics/projects. Accordingly, the study provided criteria to assess social impact based on five target groups: workers, future generations, local and national communities, international community and consumers through evaluation criteria, such as the evaluation criteria for workers which are occupational accidents, occupational diseases, wages, etc. The evaluation criteria for local and national communities are labor quality, gender equality, etc. Besides, Russell (2010) suggested that the evaluation of economic efficiency of studies should be assessed through the impact of research on national economic growth and welfare. The study also said that economically an increase in labor productivity along with the ability to provide new goods and services will increase the country's output and GDP. Antonella et al. (2012) pointed out that the socio-economic efficiency of technology research projects is expressed through several aspects: providing information/knowledge to many interested users, supporting knowledge transfer between universities/research centers and small and medium-sized industries/ enterprises, providing knowledge/ high quality information to people, improving the way that users communicate and collaborate together, facilitating social interactions, etc. In addition, in preliminary report of the European Union's Framework Programme for Research and Innovation in 2019 -Horizon 2020 (2014) about assessing the contribution of research activities, the scientific and technical aspects are assessed through the number of publications in journals with high impact indexes, the number of patents granted, the number of public -private publications, the number of times the articles cited and shared in peer-reviewed journals. Similarly, Colledge (2016) also argued that scientific effectiveness is measured by the number of downloaded studies or the coverage of studies in the mass media.
Besides, Griniece et al. (2015) conducted an assessment of the socio-economic impact of investment in research infrastructure. The paper assessed the research infrastructure in the operational phase based on aspects such as the impact on the economy, on the capacity of human resources, on the research activities, and society. In each aspect, the authors provided criteria to measure the impact based on the benefit-cost analysis method through interviewing stakeholders. Also, Geuna and Piolatto (2016) analyzed the differences between research systems in England and Italy. In the UK, the effectiveness of studies is assessed based on three aspects with evaluation weights such as research output quality accounts for 65%; the impact of research on the economy, society and culture accounts for 20%; the impact of the study on the environment and the sustainability of the study is 15%. In the case of Italy, the effectiveness of the research is also evaluated on three aspects as in the UK; however, the evaluation weights on these three aspects are the same whether they are studies in the fields of arts, humanities, social sciences, construction engineering and architecture, natural sciences and biological, including quality and quantity of output sent (50%), amount of money received from external studies (10% ), quality of recruiting and new support (10%), internationalization (10%), participation of PhD or post-doctoral students (10%); forecasting projects implemented by concessional funds (5%), improving the effectiveness of research results compared to scientific research activities (5%). In Vietnam, the evaluation of the socioeconomic efficiency of scientific research and technological development topics/ projects have also been conducted by many scientists and localities especially the recent period. However, depending on different perspectives, the way of evaluation and evaluation criteria are also different. Nguyen Ba Tra and Vo Ngoc Anh (2015) (2015) proposed assessing the socio-economic efficiency of provincial topics/ projects on five aspects, including: scientific effectiveness (including training and information); technological efficiency; environmental efficiency; economic and social efficiency. similarly, Thai Ngoc Chien (2015) also assessed the effectiveness of the application of provincial topics/projects in khanh hoa on 6 criteria including scientific efficiency, technological efficiency, economic efficiency, environmental efficiency, social efficiency and management efficiency for 6 groups of scientific fields: agriculture -fisheries, health, society -and humanities, nature, engineeringtechnology and other fields. Nguyen Truong Xuan (2012) introduced a system of criteria to evaluate the efficiency of science and technology activities in four sectors: Technology, Education, Agriculture -Forestry -Fishery and Health and Economics. Accordingly, the research has focused on evaluating the effectiveness in terms of science and training, information, technology, economics, society. At the same time, the Institute of Science and Irrigation of Vietnam (2014) also built a set of criteria to evaluate the efficiency of science and technology activities in the irrigation field through some aspects such as economic efficiency, social efficiency, scientific and technological efficiency, information efficiency, training efficiency and security and defence efficiency. In addition, Nguyen Van Tuan (2016) conducted a research on Vietnam's scientific productivity that was based on the analysis of scientific articles, the number of articles published in prestigious international journals (ISI/Scopus); linkage between local lecturers and international cooperation to assess Vietnam's science and technology productivity. However, the paper just stopped at reviewing domestic and international publications without showing the social and economic impact of these scientific tasks. Similarly, Hien (2010) based on the number of published international articles of universities, the number of citations of 11 East Asian and Southeast Asian countries assessed the scientific level of the countries in general and compared with Vietnam.
In summary, evaluating the socio-economic efficiency of scientific and technological research projects of scientists in the world and in Vietnam is quite diverse in the way of assessment as well as evaluation criteria. Economically, researchers focus on the quality of research outputs, the impact of research on the economy, labor productivity, the nation's economic growth or economic cohesion, linking research with enterprise and the commercialization of research results. Socially, studies focus on cultural and social issues, public health, social awareness, knowledge sharing. In addition, the research also addresses environmental issues, training and beneficiaries of the research results.

Research methodology
To conduct this study, we use expert method to consult 50 experts operating in some fields such as Agriculture, Natural Science, Humanities and Social Sciences, Medical Science, engineering and technological sciences about the criteria for evaluating the socio-economic efficiency of the projects after acceptance. The results of the expert survey on evaluation criteria are presented as follows: From the results of the expert survey, most experts believe that there is no need to assess the efficiency in terms of defencesecurity in provincial topics/projects. Hence, the results show that the criteria for evaluating the socio-economic efficiency of topics/projects need to be conducted on seven aspects including science, technology, economics, environment, culture and society, information -management and training. Then, we rely on previous studies and continue to survey experts on the content for each identified criterion. The expert survey results of each criterion as follow: Table 2 The results of contents need to be evaluated in each criterion

Evaluation criteria
The contents need to be evaluated 1 The efficiency in terms of science (S) The number of published articles in domestic and international journals or scientific conference proceedings from the results of the topics/projects (S1); The number of monographs are published from the results of topics/projects (S2); The number of patents granted from the results of the topics/ projects (S3); Patents from the results of the topics/projects have been applied in practice after acceptance (S4); Level of applying new knowledge from the results of the topics/projects in practice (S5); The research topic has brought new scientific points (S6); The topic has new content or new methods or new tools which was applied in the research (S7); The influence level of research in future studies (S8); The number of citations from research results in newspapers/magazines/ books / monographs (S9).

2
The efficiency in terms of technology (T) The topics/projects has given solutions regarding new technique and technologies that have applied / implemented in practice (T1); Topics/projects that has improved/created new technology which is applied/implemented in practice (T2); Topic/project that has improved/created new product which is applied/ implemented in practice (T3); Topics/projects that has improved/created new engineering facilities which is applied/ implemented in practice (T4); Topics/project that has improved/created new working process which is applied/ implemented in practice (T5); The number of technical progress, initiatives and experiences from the topics/projects are widely recognized and disseminated (T6). 3 The efficiency in terms of economics (E) Labor productivity has increased when the topics/projects are implemented (E1); The proportion of employed people has increased when the topics/projects are implemented (E2); The income of employees has increased when the topics/projects are implemented (E3); Average profit per year of the beneficiary has increased when the topics/projects are implemented compared to before (E4); the competitiveness of the beneficiaries has been improved (E5); The implementation of the project has contributed to create commercial products/goods/services (E6); The implementation of topics/projects has contributed to improve the resources efficiency (assets, capital) of the beneficiaries (E7); The implementation of the project has contributed to economic growth (E8); The implementation of the project has impacted on the change of economic structure (E9); The implementation of the project has contributed to develop a sector (E10).

Table 2
The results of contents need to be evaluated in each criterion (Continued)

Evaluation criteria
The contents need to be evaluated 4 The efficiency in terms of environment (En) Waste/pollution level can be reduced when topics/projects results are applied (En1); Fuel consumption can be reduced when topics/projects results are applied (En2); Implementing topics/projects helps managers to use resources effectively (En3); The results of topics/projecst have contributed to the conservation of biodiversity, crop and animal ecosystems (En4); when topics/projects are implemented, they contribute to improve environmental quality (En5); the creation of environmentally friendly products when topics/projects are implemented (En6); the creation of environmentally friendly new consumption way when topics/projects are implemented (En7).

5
The efficiency in terms of culture and society (CS) Increasing social welfare for people (CS1); The results of the topics/projects have a positive impact on traditional values, ethics, religion and culture (CS2); Inspiring and Encouraging creativity in the community (CS3); Contributing to conserve and develop cultural values (CS4); Contributing to increase health and quality of life (CS5); Contributing to improve knowledge and social understanding (CS6); reducing social inequality (CS7); contributing to eliminate hunger and reduce poverty (CS8).

6
The efficiency in terms of information -management (IM) Contributing to improve the management skills and information/communication administration of beneficiaries (IM1); Contributing to improve the efficiency in terms of propaganda and information (IM2); Improving the efficiency of state management for organizations implementing topic/ project (IM3); The coverage level of the research on mass media when it is deployed (IM4); Contributing to bring new information to science in general (IM5); Proposing policy implications to improve the efficiency of state management (IM6). After determining the evaluation criteria and evaluation content of each criterion, we use AHP method of Saaty (1980) to determine the evaluation weights for each topic /project. In order to use the AHP method, we conducted the construction of AHP hierarchical tree for evaluating the topics/projects following as: Next, to determine the evaluation weights for each criterion of each topic / project and the weight of each evaluation content, we determine the scale of evaluating the relative importance of the evaluation criteria according to the principle of importance ranking. For example there are 2 criteria to be assessed: A and B; The importance scale is determined as follows:

Table 3
The scale of evaluating the relative importance of the evaluation criteria There are 9 contents that need to be evaluated from S1 to S9 The results of evaluating topics / projects The process of determining the evaluation weight for each criterion and evaluation content is according to the following figure:   Fig. 2

The research results and discussion
The calculation of evaluation weight for each criterion or each evaluation content is determined by the authors based on the order of weight assignment that is presented in the research methodology. In particular, an illustrative topic / project in the Agricultural Science that has weights of the evaluation criteria as follows: Table 5 Illustrating the weights of the evaluation criteria in the Agricultural field Evaluation criteria Order assigned Weight Standardized weight (%) 1 The efficiency in terms of science 1 3 20,00 2 The efficiency in terms of technology 1 3 2 0 , 0 0 3 The efficiency in terms of economics 1 3 20,00 4 The efficiency in terms of environment 2 2 13,33 5 The efficiency in terms of culture, society 3 1 6,67 6 The efficiency in terms of information, management 3 1 6,67 7 The efficiency in terms of training 2 2 13,33 Total 15 100,00 (Source: Authors synthesize from research results) From this calculation method, the results of calculating the evaluation weight of the topics / projects in scientific fields following as: Fisrtly, in the field of Agricultural Science The above results show that the importance of criteria in terms of science, technology and economics efficiency is most concerned, followed by environmental and training effectiveness while the efficiency in terms of culture, society and information, management is considered the least important for the topics/projects in Agricultural field.
Secondly, in the field of Natural Science  The results demonstrate that the importance of criteria with regard to science, information and management efficiency is most concerned, followed by the efficiency in terms of technology and environment. By contrast, the efficiency in terms of economics, culture, society and training is considered the least important in the topics/projects in the natural science field. Third, in the field of social sciences and humanities: The results illustrate that the importance of criteria in terms of economics, culture, society, technology and management efficiency is most concerned, followed by the efficiency in terms of science and training. By contrast, the efficiency in terms of technology and environment is the least important in the topics/projects in the social sciences and humanities field. Fourthly, in the field of medical sciences: Fig. 6. Weight of each criterion for evaluating socio-economic efficiency of medical sciences topics /projects (Source: Authors synthesize from research results) The results show that the importance of criteria in terms of technology and science is most concerned, followed by the efficiency in terms of society, culture, infomation, management and training. By contrast, the efficiency in terms of economics and environment is the least important in the topics/projects in the medical sciences field. Finally, in the field of engineering and technological sciences,  The results show that the importance of criteria in terms of science, technology and training is most concerned, followed by the efficiency in terms of economics, environment, infomation and management while the efficiency in terms of culture and society is considered the least important in the topics/projects in engineering and technological sciences field. Based on the above weights, we can calculate the total score of the topics/projects by the following formulas: In which: Q: the total score of a topic / project P j : the weight of the criterion (j = 1,2, ..., 7) and is calculated as a percentage (%) Ti: the score of the criterion got according to the questionnaire. P i : the weight of component criterion ith (i = 1,2 ... n), calculated in percentage (%) The score of a topic/ project can be ranked according to the following levels: Grade A (or Excellent) includes topics / projects with the score from over 80 to 100 points. Topics / projects of this level are assessed to have very significant socio-economic efficiency.
Grade B (or very good) includes topics / projects with the score from over 60 to 80 points. Topics / projects of this level are assessed to have significant socio-economic efficiency.
Grade C (or Good) includes topics/ projects with the score from over 40 to 60 points. Topics / projects of this level are assessed to have moderate socio-economic efficiency.
Grade D (or Medium) includes topics/ projects with the score from over 20 to 40 points. Topics/projects of this level are assessed to have low socio-economic efficiency.
Grade E (or inefficient) includes topics/projects with a score of 0 to 20 points. Topics/projects of this level are assessed to have no socio-economic efficiency. It is possible that the completed project may not be transferred to organizations / individuals for use even though the project is practical.

Conclusion
The combination of expert method and Analytic Hierarchy Process method in determining evaluation weights for evaluating socio-economic efficiency of topics/projects after acceptance according to scientific sub-disciplines in Binh Dinh province which includes agricultural science, natural science, social sciences and humanities, medical sciences, engineering and technological sciences is very important. It helps Binh Dinh government to assess the socio-economic efficiency of the topics/ projects after acceptance. Thereby, Binh Dinh province has suitable orientations and investments in research activities in different fields so that they can bring high efficiency. The results show that the importance of the evaluation criteria is different in each research field. In details, for the topics/ projects in Agricultural Science, the efficiency criteria of science, technology and economics are considered the most important criteria; For topics/projects in the field of Natural Science, the efficiency of science and management information is most appreciated; For topics/projects in the field of Social Sciences and Humanities, the efficiency criteria of economic, socio-cultural efficiency and management-information are considered the most important criteria; For topics/projects in the field of medical Science, the efficiency criteria of science and technology are the most important criteria; For topics/ projects in the field of engineering and technological sciences, the criteria of science, technology and training are considered the most important criteria.