Software Engineering Competence Remote Evaluation Process Model

This research provides analysis of existing models for competence evaluation and proposes a software engineering competence remote evaluation process model. We separate competences into two domains: professional and academic. The model is developed and implemented for e-CF professional competences and academic competences evaluation for Software Engineering Master Program graduates. The remote evaluation process is based on a proposed methodology in four steps. The remote evaluation process consists of two business processes: (a) Competence test design, and then (b) Competence test completion. Business Process Model and Notation (BPMN) model was developed as a summary of both processes.


Introduction
This paper presents the results which are based on the design and implementation of the Software Engineering Competence Evaluation Internet Portal (SECEIP).The Internet portal is dedicated to joint master program training content and supervision synchronization between several European universities.The functionality of the portal provides a common support service for learning outcome information exchange, referring to a graduate's academic knowledge, skills and competence as well as to e-CF professional competences upon completion of the Master of Science in Software Engineering (Information Technology) Programs.
The rationale for undertaking this project is significant recognition of making competence-related information more accessible in the framework of the development of open educational resources (OER)-based educational materials (documents and media that are useful for educational, assessment and research purposes), which will contribute towards the development of the European Union as an advanced knowledge society in accordance with the objectives of the Lisbon strategy.So, the project aim is to foster interchange and cooperation between education and training systems within the EU (Misnevs, 2015).
The background of the SECEIP development is the following:  applying common standards for higher education qualifications across Europe in the framework of creating the European Higher Education Area (EHEA);  identifying key competences that are the basis for the employability of university graduates in the area of Software Engineering (SE);  developing and implementing an efficient web-based system that will be used for identifying and evaluating key competences of the graduates of the Master of Science in Software Engineering & Software Technologies (MS SE&ST) Programs;  making competence-related information more accessible in the framework of the development of OER-based educational materials.This paper discusses Software Engineering competence remote evaluation process model suggested within a methodology of Master Program Learning Outcome (LO) evaluation based on e-CF and augmented by academic competences (Misnevs at al., 2017).Section 2 of the paper describes related work, then Section 3 outlines the proposed software engineering competence evaluation framework, and Section 4 presents the process model proposed to take advantage of that framework.Finally, Secton 5 discusses conclusions and further work.

Related Work
The European Higher Education Area and the Bologna Process imply the implementation of active teaching-learning methodologies to allow students to develop new competences.Competence-Based Education deals with job skill training and core ability development.This approach is suitable for personalized adaptive learning, which allows learners to look for appropriate learning content based on self-competence tests (Hsu and Ho, 2012).Besides, Competence Management Systems are Human Resources Management-related systems that focus on employee life cycle, covering topics such as competence requirement analysis and personnel selection requirements (Ruano et al., 2007).
The achievement of competences has to be evaluated appropriately by means of the definition of formal and reliable evaluation processes (Montero et al., 2014).In this regard, online evaluation can be implemented securely, efficiently, and can result in increased student motivation and programming efficacy (Woit and Mason, 2003), but distributed and distance learning are creating some of the biggest changes in the evaluation process (Carter et al., 2003).The development and assessment of professional skills in higher education programs is an issue that includes defining and assessing these skills in the contexts of either a subject or an entire program (Cajander et al., 2011).Instructors prepare their students for jobs in industry, but there are many areas where these students do not possess the necessary skills or knowledge based on the expectations of employers or academia.These gaps between the abilities of graduating students and those abilities expected to have can prevent them from succeeding in their careers (Radermacher and Walia, 2013).
There are proposals to assess competences in literature, but to the best of our knowledge, there is not any initiative to assess e-CF professional competences.However, the Transferable Integrated Design Engineering Education (TIDEE) consortium of engineering educators and researchers have proposed a set of assessment instruments called Integrated Design Engineering Assessment and Learning System (IDEALS) (LeBelau et al., 2014), which can be used to evaluate professional skills such as leadership, ethics, project management and communication.IDEALS can provide evidence for demonstrating the achievement of students concerning professional skills addressed by ABET, an organization that accredit college and university programs in the disciplines of applied science, computing, engineering and engineering technology.Professional skills addressed by ABET include professional and ethical responsibility and ability to engage in lifelong learning, together with characteristics that arise through interactions with project stakeholders (Davis et al., 2010).
Academic approaches include the Platform for Adaptive and Reliable Evaluation of Students (PARES), for student testing and evaluation, which is an early web prototype that provides educators in the Greek higher education system with a tool for carrying out tests using minimal resources and for keeping the students actively involved (Kaburlasos et al., 2004).Following this line of work, Tsinakos and Kazanidis (2012) introduce a new module for PARES that eases the creation and management of question banks allowing the delivery of non-deterministically generated test suites.This PARES new module uses a conflict detection algorithm based on a vector space model to estimate the similarity between questions, thus allowing the exclusion of questions which appear to have an unacceptably large similarity from other questions appearing in the same test suite.Besides, Farias et al. (2016) show a new module for the authoring tool Easy Java Simulations (EJS) which allows automatic evaluation to virtual and remote laboratories built with EJS, allowing instructors to create interactive, remote laboratories and automatically evaluate the work of their students.
Evaluating transversal/soft competences is specially challenging.To that end Montero et al. (2014) introduce a method called Based on Teachers Reasoning (BoTR) to allow the design of automatic assessment systems for the evaluation of transversal competences of students.BoTR defines a methodology that takes into account the subjective criteria applied by teachers when assessing the considered competence to build an automatic assessment system.On the other hand, Hsu and Ho (2012) present a mobile e-learning system that considers the learners' competences throughout the learning process in order to select a learning path based on the evaluation of the students' knowledge.Llorente et al. (2014) describe the first implementation of a qualitative assessment technique based on identifying the missing competences in examination tests.Starting from this technique a quantitative evaluation is derived from evaluated weights on the evaluated and missing competencies.Moreover, Test on Line (ToL) (Tartaglia and Tresso, 2002) is a web tool that can verify the possession of ideas, the ability to combine them into deductions, and the capability to make simple numerical calculations for practical exercises.ToL generates, for each student, a proposed mark and a cognitive profile showing possible cognitional, logical, or practical weaknesses in the different topics of the study matter.Finally, in relation to Competence Management Systems, Ruano et al. (2007) present a tool to feed competence evaluation with evidences that help to evaluate the actual performance of the employees in a company.This tool allows the possibility of sharing that knowledge with the appropriate stakeholders.A formal model of competences is then proposed and a tool is discussed for both qualitative and quantitative management of competences.

Software engineering competence evaluation
A competence is often interchangeably called skill or know-how (Boucher et al., 2007. ISO 9001 (2015) defines the competence as "the ability to apply knowledge and skills to achieve intended results".The European e-Competence Framework (e-CF) (2014) defines the competence even in a broader sense than ISO 9001.The competence is "demonstrated ability to apply knowledge, skills and attitudes for achieving observable results" (e-CF, 2014).We base a software engineering competence evaluation on the definition provided by e-CF.
The objective of the e-CF is to provide a common, European tool to support organizations and training institutions in learning programs, competence needs analysis, assessment, and recruitment.The secondary goal is to provide a background to policy makers to define policies related to e-Skills development in education and in the work place.The e-CF is European Qualifications Framework (EQF, 2008) compliant; it is a suitable reference framework for competences to be dealt with as learning outcomes.
The EQF is a common European reference system aimed to linking together different national qualification systems.In practice, it acts as a translation device by helping learners and workers to move or change job across Europe.Employers and employees can use the EQF for better understanding and comparing the qualifications levels of different countries.The EQF uses eight reference levels realized through stages of education and ranging from basic (Level 1) to advanced (Level 8).Level 6 is considered to be realized through a bachelor degree, level 7 through a master degree and level 8 through a PhD degree.The reference levels are based on learning outcomes defined in terms of knowledge, skills and competences.
The e-CF is structured into four dimensions (Figure 1.).Dimension 1 contains five e-Competence areas derived from the Information and Communication Technology (ICT) business processes, enumerated from A to E. They are as follows: Plan, Build, Run, Enable, Manage.Dimension 2 refines e-Competence areas of dimension 1 into separate e-Competences.The number of defined e-Competences inside areas varies and the common set consists of 40 e-Competences.These e-Competences are general, customizable and applicable to any organization needs.
e-Competences in dimension 1 and 2 are presented from the organisational perspective as opposed to from an individual's perspective.Dimension 3, which defines e-Competence levels related to the EQF, is a bridge between organisational and individual competences.Dimension 3 defines suitable proficiency level ranging between levels e-1 and e-5 for each e-Competence.They relate to EQF levels 3 to 8. Dimension 4 lists examples of knowledge and skills embedded within e-Competences (in Fig. 1, the skill examples are listed for the e-Competence A.2).These examples provide inputs for training institutions to aid in defining of learning outcomes.In addition, they are useful in defining specific outcomes to be assessed within an organization's competence assessment programs.Dimension 4 components refer to dimension 2 but they are not related to specific competence levels in dimension 3. The key competences defined by e-CF form the basis for the employability of university graduates and they can be universally applied to graduates of all the universities of European Union in the area of software engineering.

Fig. 1. Dimensions of e-Competences
A container for the competence as a part of Learning Outcome (LO) is Master Program's subject that includes all the study activities to provide appropriate knowledge, skills and attitudes.Master program can contain at least two types of subjects: major and specialization subjects.Both types of subjects will have the same structure, however with more emphases on professional skills in specialization subjects.We provide the following competence structure (Competence = Knowledge + Skills + Attitude/Proficiency Level) as a model for the implementation in Software Engineering Competence Evaluation Internet Portal http://moodle.kic.teiep.gr/.
We separate competences into two domains: professional and academic.The professional competences must be described in terms of e-CF to show the general applicability of the subject.The academic competences are expected to be defined by the university to show the specifics of the subject.The lowest number of competences to be defined is one for each domain.The highest number is not limited.Each competence is decomposed into three parts, namely, set of knowledge, set of skills and set of attitudes/proficiency levels.Each competence item is assigned a unique name.The format of the name follows the predefined template that consists of the three parts separated by dots.The template expresses hierarchical character of the name.The first part of the name denotes a subject code (Mjfor major subjects, Spfor specialization subjects); the second part of the name represents a competence number and competence domain (Afor academic, Prfor professional); the last part of the name represents a competence part ( K -for knowledge, Sfor skills, and Pfor attitudes/proficiency level) and a number inside the competence part.For example, a name Mj01.1A.K1 denotes major subject 01, first academic competence, and knowledge item number 1.These names become the base to use for referencing.
We provide a template for subject LO description.The template identifies the knowledge, skills, attitudes/proficiency levels, and competences for the particular subject.One subject has several competences provided, but each competence also consists of several parts (knowledge, skills, attitudes /proficiency levels).The example of the template implementation is offered in Table 1.The table presents skills part of LO for the specialization subject Multimedia Methods for Mobile Devices.As we can see from the example, it is quite easy to generate academic skills; it is quite difficult to find appropriate skills in e-CF enumeration.To perform evaluation of competence, specific rubrics with evaluation requirements and grading scale are created.The descriptions of competence parts is the base for the proper rubrics development.The general template for analytic rubrics is developed on the base of the C.A.Mertler publication (Mertler, 2001).
The presented model is implemented and the subjects of Master program are available online at SECEIP portal.Each subject of the Master program has a description, defined competences, developed rubrics and test quizzes for the competence evaluation.

Remote evaluation process model
We suggest a methodology of Master Program Learning Outcome (LO) evaluation based on e-CF and augmented by academic competences.The methodology consists of the following four general steps: 1.Create a template to describe the set of competences for each dedicated subject as a collection of knowledge, skills and attitude/proficiency level.
2.Write rubrics for each item of the competence evaluation.
3.Create tests separately for knowledge, skills and attitude/proficiency level to measure each item of the competence.4.Calculate the final competence evaluation mark using an integration formula with weights.For each competence (academic or professional), a separate competence scoring rubric in the form of table is created.The rows of this table correspond to the items in the competence definition.The columns of this table correspond to the levels of scoring.Four levels of scoring are defined.They are as follows: Beginning -1, Developing -2, Accomplishing -3, Exemplary -4.Each competence rubric is labeled using the competence identifier (e.g.Mj10.1A or Sp02.1Pr).
Each competence rubric contains three parts: knowledge, skills and attitude/proficiency consisting of several items.For each item in the competence definition, a separate item in scoring rubrics is developed.Total competence score calculation is performed using weights from 0 to 1 (e.g. for the academic competence part Knowledge -0.5, part Skills -0.3 and part Attitudes/Proficiency-0.2, and for the professional part Knowledge -0.3, part Skills -0.4 and part Attitudes/Proficiency -0.3).
Instructors define rubrics for their subjects as described above.Once scoring rubrics are defined, students can assess their competences remotely and automatically through the SECEIP portal.Thus, the remote evaluation process model can be divided into two business processes: (a) Competence test design, and then (b) Competence test completion.BPMN (OMG, 2011) models show a summary of both processes in Figure 2.

Conclusions
The implementation of the remote evaluation process model for promoting the Open Educational Resource-based educational materials will help making key educational resources sharable, storable, findable and interoperable on a global scale.Using a common online format for describing, referencing and sharing the graduate learning outcome definitions described in the Program will make it easier for educators to assess the compatibility of educational systems and employment sectors across national borders in the framework of creating the European High Education Area.By managing and sharing this competence evaluation data, students will be able to better plan their careers and enhance their employability potential.

Table 1 .
Competence decomposition into parts