Evaluation of the indicators of professional public accreditation of agricultural study programs

The article discusses the results of the analysis of the accreditation indicators used in the quality assurance systems of Russia and Europe. The indicators of professional public accreditation of agricultural programs were developed based on the Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG) within the TEMPUS Project «Development of Public Accreditation of Agricultural Programs in Russia» (PACAgro). The indicators provide for the implementation of requirements and employers’ participation in the evaluation of study programs. The article describes a list of indicators and introduces the methods for their assessments. A system for the indicators’ assessment when conducting external and internal review during professional accreditation of study programs was developed based on the scale grading from 0 to 3 points.


Introduction
In the Russian Federation the labor market impact on the system of higher education has been until recently limited; there was a significant gap between the content of education, which was established by the Ministry of Education of Russia, and employers' needs.In 2013 in light of the higher education system reform the Law on Education in the Russian Federation was adopted.It prescribed greater involvement of employers in the work of higher education institutions (HEIs), as well as in the system of external quality assurance of education (Self-Evaluation Report of the National Accreditation Agency, 2015).The legislation introduced the concept of "professional public accreditation", which enables employers to significantly influence the content and quality of education, to establish their own requirements, which makes the graduates of HEIs with such accreditation more competitive in the labor market.As part of this concept, the system of professional public accreditation of the agrarian sector in the Russian Federation was created and is being implemented in practice within the TEMPUS Project (Development of Public Accreditation of Agricultural Programs in Russia, 2016).
One of the main challenges here is to develop a list of indicators that allows for adequate and objective assessment of study programs in terms of their compliance with the requirements established in the framework of accreditation.Indicators used in any system of accreditation of study programs are its basic element, and determine the content of internal and external evaluation (Petropavlovskiy, M., Smelik, V., Nefedova, O, 2015). http://dx.doi.org/10.15414/isd2016.s8.15

Data and Methods
The indicators of professional public accreditation of agricultural programs were developed with due regard to the approaches defined by the Standards and Guidelines for Quality Assurance in the European Higher Education Area, (ESG) 2015.In addition, the analysis of the indicators used by the agencies carrying out accreditation of study programs in Spain (Catalan University Quality Assurance Agency, AQU Catalunya), Germany (Agency specialized in accreditation of engineering programs, ASIIN), as well as some Russian accreditation agencies (Association for Engineering Education of Russia, OPORA RUSSIA) was conducted.
The results of the analysis shown in Table 1 allow for the conclusion that the majority of Russian and foreign agencies use indicators (or accreditation standards), which can be grouped as follows: 1) study program; 2) students and graduates; 3) academic staff; 4) facilities and resources; 5) organization and management.The developed indicators should take into account the requirements of employers, as in accordance with the Russian legislation "professional public accreditation of professional study programs is the recognition of the quality and level of training of graduates who have mastered such a study program in a particular organization, carrying out educational activities, to meet the requirements of professional standards, the requirements of the labor market to specialists, workers and employees of the relevant profile".

Content of the Indicators of Professional Public Accreditation of Agricultural Study Programs
It  participation of employers in the educational program development and updating: the study program is regularly updated.The improvement of curricula, syllabuses of disciplines (modules), practice and research programs, state final examination are carried out with the involvement of employers.There is a system of interaction between an educational organization and employers, labor market representatives and other organizations (agricultural enterprises, organizations, agribusiness authorities, suppliers of goods and services of the agricultural sector).
 structure and content of the study program curriculum provide for the achievement of competences and conferred qualifications: the study program trains students for effective and responsible work in the profession in accordance with the labor market needs.The implementation of the study program curriculum ensures the achievement of the established competencies and conferred qualifications.
 system for assessing learning outcomes: procedures for assessing the level of students' knowledge/competences are documented, based on clear criteria and designed in accordance with the intended learning outcomes.The fund of assessment tools meets the requirements of professional standards developed and approved by the Ministry of Labor of RF as regards employment functions, actions and labor-market skills.The results of students' technical training are assessed by employers during professional certification.
 -efficiency of the applied teaching methods: the applied teaching methods are well grounded and contribute to the achievement of the established competencies.The educational process includes methods of on-line training aimed at developing student professional competences (technology for solving professional tasks, lectures-visualizations, problembased lectures, training of professional competences, information communication technologies and etc.).
 teaching and learning support of the study program: teaching and learning support is focused on the established learning outcomes, corresponds to the required quantity and quality.Teaching and learning aids are developed with the involvement of employers and/or have a critical review of employers.
 graduate qualification works, scientific research projects: the theme-subject of graduate qualification works and scientific research projects allow for assessing the level of formation of professional competences.Requirements for their implementation are consistent with the level of competences and qualifications conferred.Questions and tasks are designed to meet the specific requests of employers.
 providing the implementation of curriculum and its improvement: the monitoring of the curriculum implementation is systematically carried out when delivering the study program, the monitoring results are used for its improvement.

Indicator 2. Students and graduates
 level of knowledge /competences of students and graduates: the level of knowledge/competences meets professional standards.
 student support services: students have the opportunity to use student support services, including academic advising and counseling, as well as the existing career guidance services aimed at helping students in the right career choice and employment at agricultural enterprises.
http://dx.doi.org/10.15414/isd2016.s8.15  employment of students: there is a system for tracking professional career paths of graduates.The reliable data on students' employment at agricultural enterprises over the past three years is published at the web site of the educational organization.Level of graduates' employment.
 satisfaction of employers: the proportion of partnersemployers satisfied with the results of training (measured by the results of surveys conducted among employers of agricultural enterprises).
 satisfaction of students: the proportion of students satisfied with the actual learning outcomes and the work of student support services (measured by the results of surveys conducted among students studying the program under accreditation during external review).Indicator 3. Academic staff  compliance of the academic staff qualification and number to the curriculum: academic staff (hereinafter -the AS) have the education appropriate to the taught disciplines.The proportion of teachers with academic degrees and titles, as well as professional experience in the area specific to the taught discipline, meets the requirements set by the established standards.
 involving employers in teaching: representatives of large agricultural enterprises are engaged in teaching.The educational organization provides for their instruction, guidance, monitoring and evaluation of their activities.
 professional development of teachers: teachers undergo professional development training not less than once in 5 years at modern agricultural enterprises, scientific research institutes of the agrarian sector, participate in agro exhibitions, conferences, symposia, and so forth.
 availability of the system for diagnosis and motivation of the academic staff: the educational organization has a system to stimulate AS.The quality of each teacher's performance is regularly evaluated, and the results are used to motivate AS.

 provision of the educational program with classrooms, laboratories and equipment:
the educational program has an adequate number of classrooms, laboratories and equipment in accordance with the curriculum requirements.
 improvement and enlargement of facilities and resources: facilities and resources are constantly updated and improved in accordance with the requirements of the agrarian sector of economics.
 engagement of employers in providing the necessary base for the implementation of the educational process: employers take part in updating facilities and resources of the educational organization.Employers provide their premises for training and scientific practices of students.

Indicator 5. Organization and management
 management of the study program: the organizational structure of the program management is effective.The management process is aimed at improving the study program.Teachers, employers and students are involved in the decision-making process in accordance with their areas of responsibility.
http://dx.doi.org/10.15414/isd2016.s8.15  availability of the system for collection, analysis and use of the information when managing the educational program: there is a system of key performance indicators in relation to the departments responsible for the program delivery.The opinion of employers of agricultural enterprises, students and AS is taken into account when evaluating the performance of departments.

Assessment of the Indicators of Professional Public Accreditation of Agricultural Study Programs
When conducting external review it is suggested to use a four point grading scale for each criteria unfolding the indicators' content.For example, in order to assess the employers' involvement in the development and updating of the study program (Indicator 1. Study program) the following grading scale is used: 0 pointno documents proving the availability of the study program are presented; 1 pointthe study program is developed and annually updated; 2 pointsthe study program has a referee report or reference from at least two leading employers of the region; 3 points -availability of the document describing the program development strategy for the next 4-6 years; program development strategy is line with the prospects of development of the regional (federal) labor market, trends within the areas of agricultural program training; the study program is developed in conjunction with the leading agricultural organizations in the region or has references/referee reports from them; syllabuses of study disciplines (modules) have expected learning outcomes agreed with the key partners representing the regional labor market.
Teaching and learning support of the study program (Indicator 1) is analyzed in terms of the following: -teaching and learning aids are developed for all disciplines, practices, types of classroom hours and etc. in accordance with the program structure and content and can be used in all education delivery modes realized within the program; -there are mechanisms for updating teaching and learning aids with regard to the changing conditions at federal, regional and local labor markets; -teaching and learning aids are available for students in the internal information system; -teaching and learning aids are handy and encourage students for further self-education and independent work; -teaching and learning aids are developed with the involvement of employers-organizations interested in graduates' employment.
Objective information on graduates' employment (Indicator 2) is posted on the monitoring web site of the Ministry of Education and Science of RF (http://graduate.edu.ru/) and may be used as one of the information sources for the assessment of the corresponding criteria of the indicator.
Level of knowledge/competences of students and graduates (Indicator 2. Students and graduates) is assessed by the results of their professional certification expressed in a percent of students and graduates who has been awarded a corresponding professional certificate. http://dx.doi.org/10.15414/isd2016.s8.15 In order to assess the level of students and graduates' competences, employment of graduates of the study programs, satisfaction of employers and students it is suggested to use a quantitative grading scale as well: 0 pointindicator value from 0 to 70%; 1 pointfrom 70 to 80%; 2 points -from 80 to 85%; 3 pointsindicator value over 85%.
The final assessment of each indicator as a whole is formed by summarizing the points given by experts with regard to all content criteria.The final indicator assessment is positive if it constitutes over 50% points for all criteria:

Conclusion
A list of indicators for carrying out professional public accreditation of agricultural programs at HEIs has been developed within the study framework allowing for the employers' influence on the content and quality of education.In order to assess the indicators a scale from 0 to 3 points has been introduced and a methodology of peer review for each indicator has been developed.It was suggested to use the model of integration of the indicators' assessments for making a decision on accreditation or refusal of accreditation of the study program being evaluated. http://dx.doi.org/10.15414/isd2016.s8.15

Table 1 : Comparison of the indicators used by Russian and foreign accreditation agencies
is suggested to use the following set of indicators when carrying out Professional Public Accreditation of Agricultural Study Programs.

Table 2 : Number of criteria and points for the assessment of Indicators of Professional Public Accreditation of Agricultural Educational Programs
Positive decision on accreditation of the study program within professional public accreditation of agricultural programs is possible when all indicators are positively assessed, conditional decisionwhen 4 indicators have positive assessment, in all other cases the program is considered not accredited.The assessment criteria and decision rules are suggested to be tested in2016during accreditation of study programs delivered by Russian HEIs-project partners: Saint-Petersburg State Agrarian University, Russian State Agrarian University -Moscow Timiryazev Agricultural Academy, Kabardino-Balkarian State Agricultural University named after V.M. Kokov, Kazan State Agricultural University, Mari State University.