The Comprehensive Evaluation of Electronic Learning Tools and Educational Software ( CEELTES )

Despite the fact that digital technologies are more and more used in the learning and education process, there is still lack of professional evaluation tools capable of assessing the quality of used digital teaching aids in a comprehensive and objective manner. Construction of the Comprehensive Evaluation of Electronic Learning Tools and Educational Software (CEELTES) tool was preceded by several surveys and knowledge obtained in the course of creation of digital learning and teaching aids and implementation thereof in the teaching process. The evaluation tool as such consists of sets (catalogues) of criteria divided into four separately assessed areas – the area of technical, technological and user attributes; the area of criteria evaluating the content, operation, information structuring and processing; the area of criteria evaluating the information processing in terms of learning, recognition, and education needs; and, finally, the area of criteria evaluating the psychological and pedagogical aspects of a digital product. The specified areas are assessed independently, separately, by a specialist in the given science discipline. The final evaluation of the assessed digital product objectifies (quantifies) the overall rate of appropriateness of inclusion of a particular digital teaching aid in the teaching process.


Introduction
Distinguishment of a high-quality educational product from other, less appropriate products, or even socially dangerous electronic didactic means, is an exceptionally difficult task.It requires determination of quality standards and existence of special evaluation tools respecting the specific features of electronic teaching and learning aids and software applications with the ambition to educate (Karolčík et al., 2013).Despite the fact that there are relevant opinions which regard any strict limitations put to the educational software group as dangerous and inappropriate (Kalaš, 2005;Lever-Duffy and McDonald, 2011;Foshat and Ahmed, 2003;Resnick, 2002), majority of the published studies accept the inclusion of a software product developed to support the education among special products which are subject to special requirements (Jeff, 2013;Kalaš, 2005;Lehotská, 2007;IT232 -Pedagogical software, 2011;Educational software, 2014).Unfortunately, production of multimedia learning materials is frequently not informed by relevant research in psychology and education.Therefore, the Internet offers a large number of completed digital educational objects of various quality levels (Leacock and Nesbit, 2007;Nesbit and Leacock, 2006;Shavinina and Loarer, 1999).

Digital Teaching/Learning Aid Quality Assessment
In times of general digitalisation of public and private life of the society, the objective and complex evaluation of digital teaching aid quality represents a great challenge for the current pedagogy and psychology.
Attempts to identify key properties of software applications with regard to specific education requirements date back to the times of experimenting with software in schools and gradual integration thereof in the teaching agenda.At the turn of the millennium, evaluation tools were mostly in form of simple forms containing scaled items grouped into several categories (e.g.Content, Assessment, Technical Quality) (Kelly, 2002).They were intended for teachers.In their structured layout, teachers were evaluating basic characteristics of a particular used digital teaching aid and describing its strengths and weaknesses.Concluding parts of the form were focused on the summary of the acquired pedagogical experience and the description of teaching procedures and strategies, in which the use of a particular teaching aid brought good learning results.Exponential rise of the number of education-focused software applications and electronic teaching materials brought the need of categorising, grouping and dividing them.
In an effort to mitigate the problem of searching the appropriate software, some information sources offer their own quality evaluation metrics, which is applied to a specific digital product and the measured values are then reflected in the search requirements (Vargo et al., 2003).However, the efficacy of this technique of searching the quality teaching aids is directly dependent on applicability and appropriateness of the selected evaluation tools that generate the quality ratings.
Evaluation of the quality of special teaching aids of the digital nature is discusses also in the part of the book named Evaluation and Selection of Learning Resources: A Guide (2008).Authors classified video files on DVDs (DVD/Video), digital software products (Digital Resources) and sources available on the web (Web Resources) as such teaching aids.Individual evaluation tools were grouped into four main categories: Content, Educational Aspects, Technical Aspects, and Social Issues.They are further divided into individual criteria groups and subgroups.As for the digital teaching aids, the evaluation criteria groups in categories of education and technical aspects are extended with additonal assessed characteristics, such as simple navigation in the program environment (User navigation through program is appropriate), interaction-supported meaningful learning (Interaction promotes meaningful learning), possibility to integrate the program in other teaching activities within the subject (The program can be integrated with other activities in the same subject), immediate, positive and motivational nature of the feedback (Feedback is non-threatening, immediate, positive, motivational, and user-sensitive), and as for technical aspects, for example availability of the help function (A help function is provided and appropriate), or the possibility for a teacher to control a student's progress and achieved results (Teacher control of feedback and progress is provided and appropriate).Annexes to the materials also include three separate evaluation forms by Kathleen Schrock, particularly created for the evaluation of quality of a website with the educational content (Critical evaluation of a web site) from the teacher's point of view and from the point of view of educational needs in the first and second levels of basic schools.(For use by educators, Elementary school level and Secondary school level).
Criteria catalogues were developed for purposes of evaluating educational software products in the German-speaking environment.The authors of this study among other things stress the fact that the process of the evaluation of educational software is often marked by subjectivity and that is why the purpose of the evaluation of the software (Learnsoftware) is not its inclusion into the category good or inappropriate.They also stress the importance of the following function of the existence of the catalogue of evaluation criteria which should improve creation of new educational software mainly from the pedagogical point of view (Gottfried et al., 2002).
The thought-out and well-elaborated methodology of the digital learning object evaluation (The Learning Object Review Instrument -LORI) is presented by the study by authors Leacock and Nesbit (2007).Digital learning objects are small, simple, modular and discrete units designed especially for electronic delivery and use (What are Digital Learning Objects?, 2014).Authors of the study (Leacock and Nesbit, 2007) realize that the described learning object review instrument (LORI) is not sufficient tool for the comprehensive evaluation of the quality of educational programs consisting of a large number of individual learning objects.
Evaluation of multimedia, pedagogical and interactive software in the context of the teaching and education process represents the topic of the study by French authors (Crozat, S., Hû, O., Trigano, P.) which characterises in more details the EMPI method (Evaluation of Multimedia, Pedagogical and Interactive software) (Crozat et al., 1999;Crozat et al., 1999).They point out the fact that the general precondition of elaboration of a usable evaluation method is to know the answer to 3 important questions: Who is the evaluator, what is the subject of the evaluation, and when it is evaluated (Depover, 1994).In case the EMPI method is used, the evaluators will be the users of multimedia software, i.e. experts in pedagogy, didactics, methodologists, or instructors working in education centres.The evaluation only applies to finished software products and the proposed evaluation system divides the evaluation criteria into 6 separate modules.For each of them, special evaluating forms were created.

Methodology for the Creation of the Evaluation Tool Concept
Elaboration of the concept for the comprehensive evaluation of educational software and electronic learning materials was preceded by the performance of two survey projects.The methodology applied within the initial survey was based on the Delfi method.In 2013, questionnaires with a special structure, containing exclusively open items, were administered to 3 groups of addressed experts who regularly use digital technologies in their usual working or private life.These experts provided their statements regarding 3 issue areas related to the expected most important properties of software applications and electronic materials especially designed for the education needs and teaching support.The survey brought rather definite results and the conclusion that the simplicity and lucidity of installations, launch and orientation belong to the most significant properties of education-based software products (Karolčík et al., 2013).The second survey was carried out with the sample of 1,842 teachers -graduates of specialized education in the field of efficient and reasonable use of digital technologies in the educational process that was carried out within the National Project of Modernization of Educational Process at Elementary and Secondary Schools (MVP, 2014).All the addressed respondents successfully passed their courses by the elaboration of the final thesis, defence thereof, and the final exam.Thereby, they proved an adequate level of knowledge about a suitable and methodically well-prepared application of digital technologies in the teaching process.They can thus be regarded as experts in this area and their opinions on the issues of digital learning aids quality evaluation as substantial and important.The used questionnaire with a specialised structure contained closed questions with multiple answer options and scaling items.The survey was exploring the opinions and attitudes of teachers regarding the issues of significance of digital learning aids quality evaluation by independent experts and the optimal method of publishing the achieved results, including the methodology of the appropriate use of the aids in education.The questions with multiple answer options inquired about the preferences among teachers with regard to the most important properties of software products with the ambition to educate.The research confirmed, inter alia, that teachers regard as exceptionally important the knowledge of the quality of available digital teaching aids and such information, supplemented with suggestions, methodologies, and recommendations for didactically suitable use of a particular aid, would be very helpful for them in their work.However, they expect that the evaluation will be elaborated by experts -specialists with required qualification, who perfectly understand the issues (fields of science) in which they evaluate a product.The summary results from the questionnaire survey, including the detailed statistical analysis of the measured data, will be processed in a separate scientific study.
Our concept of the Comprehensive Evaluation of Electronic Learning Tools and Ed-Comprehensive Evaluation of Electronic Learning Tools and Ed-Comprehensive Evaluation of Electronic Learning Tools and Educational Software (CEELTES) uses the knowledge of the existing evaluation tools and published experience with their use or development.It considers important conclusions and observations from our research.Unlike majority of known evaluation concepts, it minimises the rate of subjective freedom of an evaluator when assessing the determined criteria and maps more precisely (in more details) all the specified key characteristics of a particular digital teaching aid.The risk connected with incorrect or imprecise un-derstanding of the content of the specified criteria by the evaluator is minimised here by the choice of experts -specialists in the given field of science.Each of them evaluates a digital teaching aid always with regard to a single evaluation topic.
Extraordinary width and variety of software products usable in the teaching process complicates, to a large extent, the implementation of unified and simple patterns for their evaluation.This problem is solved by authors of evaluation tools by elaboration of several questionnaires which always evaluate a single type of a digital teaching aid.Our concept of the Comprehensive Evaluation of Electronic Learning Tools and Educational Software (CEELTES) offers a simple model of the evaluation methodology.An evaluator selectively chooses from the range of criteria only those, for which their applicability or inapplicability can be assessed (assumed) for a particular type of a digital product.
As it is not possible to use the LORI as a general instrument for the evaluation of digital learning aids and the EMPI method does not take into account extraordinary diversity and variety of available technological solutions, we based our proposal of the evaluation areas definitions and specification of individual criteria on the group of complex learning tools.These learning tools represent the most complex, very thoroughly prepared and elaborated software solutions designed especially for the teaching and learning purposes.They mostly integrate also other groups of products, such as instructing programs, applets, special software modules or complements, into a single functional unit.Particularly an extraordinary wide range of the offered educational content, a wide range of various functions, settings, and special instruments for the system administration, enable the assessment of the largest number of product quality aspects (Karolčík et al., 2013).

The Comprehensive Evaluation of Electronic Learning Tools and Educational Software (CEELTES)
The elaboration of the concept for the comprehensive evaluation of the quality of educational software and electronic learning materials was based on five key requirements: The key requirements were divided into four separate areas: The area of technical and technological attributes. 1.
Information processing in terms of learning, recognition and education needs.

4.
The evaluation areas consist of the sets (catalogues) of evaluation criteria and result quantification proposals.Each evaluation area is assessed by a qualified expert -a specialist in the given area.This minimises the incorrect or inadequate understanding of the content of the determined criteria.Practical use of the proposed point scale is complicated by a high diversity of the existing software products.Therefore, quality assessment with a larger group of criteria within the assessed area requires also a summary evaluation of the measured properties of electronic learning aids.For the same reason, an evaluator does not have to use all the defined criteria and can choose only those, applicability or delivery of which can be objectively assessed in a particular digital product.Criteria, for which the point assessment is not possible, are not included in the overall area evaluation.Every determined area is evaluated separately and autonomously without identifying a precise proportion of possible participation in the overall evaluation result.
The summarising and complete list of assessed criteria included in the concept of comprehensive evaluation of the quality of educational software and electronic learning material is presented in separate chapters.It is also included in the electronic content of the website http://www.evaluedu.sk/sk/home/evaluation-of-electronic-learning-tools-and-educational-software/.The article presents only a fundamental structure of the comprehensive evaluation instrument and examples of the criteria in selected evaluation areas.Maximum number of points assignable to an assessed learning aid in a particular criteria area is shown in brackets.

Technical and technological attributes
Any high-quality, methodically well prepared and excellently processed electronic content of a digital teaching aid ceases to be educationally attractive, if technical problems disable to use it in the teaching process.Technical and technological parameters of a digital teaching aid are not directly related to the quality of processing of the presented educational content.However, they significantly affect the extent of possible application of the aid in the educational practice.Moreover, structured information about technical and technological attributes of a digital teaching aid has also an informative function, as it notifies a user about the limitations of a selected technological platform and the requirements for the technical equipments in schools.

Defined groups of evaluation criteria:
1.a Installation and launch (23 points) Application acquisition does not require formal registration of a user

Content, operation, information structuring and processing
Due to its nature and focus, the second group of the evaluation criteria is partially inspired by the key requirements related to textbooks and educational texts.This group is supplemented with the lists of criteria that evaluate technical and technological limits of a selected program platform used in the presentation of the educational content.

Defined groups of evaluation criteria:
2.a Content (50 points) Software content respects the fundamental pedagogical documents -national cur-• riculum, educational programs, content and performance standards.
Information processing by the software develops the key competences of a learner • prescribed in the applicable and binding pedagogical documents and the information is presented in a logical structure.Software content is completely localised into the particular mother tongue (na-• tional language).Software content provides the themes for educational projects, problem tasks and objective manner and in compliance with the respect for human rights and freedoms.
Detailed definitions were determined also for the groups of criteria evaluating • the content, operation, information structuring and processing in the following areas: 2.b Operation (16 points) 2.c Information structuring and processing (14 points)

Information processing in terms of learning, recognition, and education needs
If we regard didactic means to be any sources of material, as well as non-material nature, which facilitate the fulfilment of the determined educational objectives, the educational software product represents one of the most up-to-date forms thereof.Therefore, it is particularly important that a digital teaching aid serves also as motivation for users, provides them with quality feedback, and enables them, in a required extent, to control their learning or teaching.Authors of a digital teaching aid must also provide a high-quality professional methodical assistance and didactic support.This purpose would be best served by practical examples of teaching activities, including procedures and suggestions how to integrate, in a proper and meaningful manner, the educational software in the teaching agenda.Our research confirmed that particularly the examples how to use a digital teaching aid are regarded by teachers as the most usable in their pedagogical practice.

Defined groups of evaluation criteria:
3.a Didactic information processing (24 points) Thanks to the diversity and variety of the content, the software enables a teacher • to use a wide range of teaching methods and procedures.
Software is open and assumes the provided information database will be subse-• quently used in the learning process and does not hinder that.Software can be used in all stages (sections) of a lesson.Hypertext links refer to other information sources and the related facts are high-• lighted in the text block and easy to distinguish.Detailed definitions were determined also for the groups of criteria evaluating the • information processing in terms of learning, recognition and education needs in the following areas: Educational activities focused on a meaningful, logical organizing of terms, ideas, • and visual components mediated by the learning software (active information processing, structuring it into a comprehensive understanding of the acquired contents, which is demonstrated by deducting meaningful, logical relations, applications, etc.), asking verification questions to verify the level of meaningful learning.
Learning activities focused on presenting examples of acquired terms, related de-• tails, relations to other terms and concepts that were the subject of previous learning, creation of more complex interdisciplinary relations and applications of the curriculum content in everyday life conditions, verification questions and tasks to verify the aforesaid relations and applications.
Requirements for active information processing and interpretation, using the • previous knowledge and thinking (e.g.formalizing the structures of one's own knowledge in form of term maps, thinking about consequences of manipulations, changes in the environment on the level of thought experiment, manipulations in the virtual environment, etc.).
Possibility to solve tasks and problems as the source of new knowledge, i.e. ap-• plication of learning on the individual or the group levels using the cooperative learning.
Detailed definitions were determined also for the groups of criteria evaluating the • psychological and pedagogical aspects in the following areas:

Overall evaluation of a digital learning aid
The final, complex evaluation of the assessed educational software (electronic learning aid) includes the number of assigned points in individual evaluation areas (rough score) and the percentage representing the ratio to the maximum possible number of the assigned points.These indicators are supplemented with the verbal evaluation divided into three categories.Verbal evaluation degrees take into account the fact that we work with a strict model of evaluating the quality of educational digital product that compares the existing software products with the ideal model.Therefore, exceeding 50% of the point score limit in 3 evaluation areas can be regarded as a very good result and the evaluated digital learning aid as a very good tool.An excellent digital learning aid is characterised by at least 80% of the assigned points in 3 out of 4 evaluated areas and good aid in case of more than 30%.A software product that failed to obtain at least 30% of the total number of points in majority of the evaluation areas, as for the quality evaluation, cannot be, in our opinion, regarded as educational, even though its individual use does not exclude interesting educational results.

Conclusion
The Comprehensive Evaluation of Electronic Learning Tools and Educational Software (CEELTES) represents a special and modular evaluation instrument for the assessment of digital learning aid quality.The system consists of 4 evaluation areas consisting of a set (catalogue) of evaluation criteria and the interval quantification of the achieved results.The preparation and development of the system was preceded by several research projects.They mapped the extent and nature of the available software solutions, as well as the properties expected from the software applications and electronic materials especially intended and designed to support learning by the users of digital technologies.

Annexes
The

Area -Content, operation, information structuring and processing
Evaluation criteria * (* If a digital learning aid meets the criteria in the given sub-area, it is assigned the maximum number of points shown in brackets!In case it meets the criteria only partially, the score is adequately reduced.If the evaluator is not able to assess objectively the applicability or inapplicability of any of the determined criteria, such criterion is skipped and the respective maximum number of points shown in the brackets will not be included in the overall score.) Software focused on science branches focused on the Earth) Software enables a user to move over the interactive map, where the marked locations are interconnected with the corresponding information database (0-2 points)

2.c.1
Software structure is well-organised, logical, and thoroughly thought-out.


Orientation inside the structure is easy for users (0-2 points) Information is arranged in blocks, graphically clearly divided, segmented into

Area -Information processing in terms of learning, recognition, and education needs
Evaluation criteria * (* If a digital learning aid meets the criteria in the given sub-area, it is assigned the maximum number of points shown in brackets!In case it meets the criteria only partially, the score is adequately reduced.If the evaluator is not able to assess objectively the applicability or inapplicability of any of the determined criteria, such criterion is skipped and the respective maximum number of points shown in the brackets will not be included in the overall score.)

3.c.1
Software is supplemented with a printed or electronic handbook, user manual, or  a video, a course with tips, ideas on how to use and work with the software (0-4 points) Software has a special website containing the information and the methodical  recommendations how to work with the software at the lesson, or in which education stage the use thereof brings the best educational effect and results (0-4 points) Software website is regularly updated and supplemented with new didactic  materials, work sheets, tasks, and practical experience of users with the software application at the lessons (0-6 points) Teachers can actively participate in the production of methodology, projects,  educational activities contained in the software website (0-2 points) 16 Altogether for area 3 60

Area -Psychological and pedagogical aspects
Evaluation criteria * (* If a digital learning aid meets the criteria in the given sub-area, it is assigned the maximum number of points shown in brackets!In case it meets the criteria only partially, the score is adequately reduced.If the evaluator is not able to assess objectively the applicability or inapplicability of any of the determined criteria, such criterion is skipped and the respective maximum number of points shown in the brackets will not be included in the overall score.)

4.a.1 (ISCED 1)
Examples of terms, processes, and relations are presented in a demonstrative, illustrative,  figurative form (e.g.pictures, demonstrative schemes, realistic video sequences, simulations, animations, etc.) (0-8 points) Tasks including data processing and results presentation in a graphical form, (

4.b Taking into account the activity level in the learning activities
Learning activities focused on a meaningful, logical organizing of terms, ideas, and visual  components mediated by the learning software (active information processing, structuring it into a comprehensive understanding of the acquired contents, which is demonstrated by deducting meaningful, logical relations, applications, etc.), asking verification questions to verify the level of meaningful learning (0-4 points) Learning activities focused on presenting examples of acquired terms, related details, relations  to other terms and concepts that were the subject of previous learning (0-3 points) Creation of more complex interdisciplinary relations and applications of the curriculum content  in everyday life conditions, verification questions and tasks to verify the aforesaid relations and applications (0-2 points) Requirements for active information processing and interpretation, using the previous  knowledge and thinking (e.g.formalizing the structures of one's own knowledge in form of term maps, thinking about consequences of manipulations, changes in the environment on the level of thought experiment, manipulations in the virtual environment, etc.) (0-3 points)

15
Possibility to solve tasks and problems as the source of new knowledge, i.e. application of  learning on the individual or the group levels using the cooperative learning (0-3 points)

4.c Taking into account the constructiveness in learning activities
Learners can verify the degree of curriculum content acquisition in terms of understanding it in  relation to the previous learning and knowledge (e.g. in form of tasks to supplement the term map with other components and describe the relations between the components, solve the tasks to interpret the relations between the terms, concepts, ideas, formulas, etc. (0-3 points) The feedback is sufficiently frequent, in terms of understanding the curriculum content it is  instructive and induces higher cognitive processes of learners (applying creativity, critical thinking when considering and solving a problem, self-regulation -possibility to verify the level of one's own knowledge, setting the learning objectives, selection of learning procedures, etc.) (0-3 points) Possibility to solve tasks and problems that reveal basic explanatory concepts and mechanisms Opportunities for learners to attend to learning and game-based educational activities which are  attractive for them, after they manage the compulsory learning activities (0-2 points) While solving more complex tasks, in irregular intervals (depending also on a learner's previous  success) the software evaluates the correctness of the procedure; in case the procedure or the solution is correct, it provides substantiation; in case it is not correct, it provides a learner with the guidance for further learning or a procedure (0-3 points) An alternative in solving more complex tasks (objectives) is their segmentation in partial  tasks (objectives); an optimal situation is when the segmentation takes into account individual differences in learners' knowledge, skills, and abilities; work results corresponding to the objectives and tasks are then evaluated by the software for correctness and solution requirements and the information on a subsequent procedure is presented (0-2 points) Software enables learners to learn how to distinguish between various stimuli, objects,  processes, terms, concepts (e.g.various species of plants and animals, chemical reactions, mental functions, etc.), provides learners with the distinguishment principles (or distinguishment signs), assigns tasks containing distinguishment and provides the correctness feedback (0-3 points) Software enables learners to learn how to generalize various stimuli, objects, processes, terms,  concepts (e.g.learners search for common signs of particular object groups, create categories and subsequently define them; learners think about common signs of different situations and the method of responding to these situations, etc.), it can also provide the learners with the principles to be applied in generalization, assign the generalisation tasks, after each generalization provide the learners with the correctness feedback (0-3 points) 20

4.h Taking into account the requirements following from the cognitive architecture and its function (applying the cognitive learning theory)
Simplicity, intuitiveness of learning activities in the software operation,


When providing the information in electronic form, use a verbal and visual form thereof

4.k Taking into account individual differences depending on the profile of multiple intelligences
Content variety, diversity of learning activities of learners using their language skills and  logical and mathematical skills, spatial imagination, physical skills, social abilities and skills, self-knowledge and self-regulation and occasionally also musical abilities and skills (0-3 points) Assign tasks that stimulate learners to perform various activities (self-study, social interaction,  performance of manual or other physical actions, etc.) (0-2 points) 5

4.l Taking into account individual differences following from different types of preferred learning motivation of learners
For learners with prevailing performance motivation -provision for the feedback that enables  learners to evaluate and compare their performance from different points of view, set the program parameters (various difficulty levels, various time intervals for the work with the program) (0-2 points) For learners with prevailing cognitive motivation -provision of hypertext links to extended the  curriculum content, provision of various problems and creative tasks, solution of which satisfies the cognitive interests and enriches their knowledge and skills (0-2 points)

5
For learners with prevailing social motivation -possibility within the program to cooperate,  communicate with other persons (teacher, co-learners, parents, experts, etc.), possibility for other persons to obtain the feedback on a learner's results, as well as the possibility for learners to compare themselves with other learners in terms of the curriculum content acquisition (0-3 points) Altogether for area 4 160

3. b
Motivation and activation of learning activities (20 points) 3.c Methodical support (16 points) 4. Psychological and pedagogical aspects Defined groups of evaluation criteria: 4.a Current level of cognitive development of learners and stimulation of their cognitive development (24 points) For the learners in the secondary stage of elementary schools they provide the following: Opportunities to consider possible, hypothetical, and unreal situations and pos-• sibilities and also the consequences of their update (solution of tasks with unreal input data).Complex tasks requiring systematic solution procedures, search through the pos-• sibilities, creation of strategies, probability-based thinking.Mediated information requires understanding, interpreting of, and thinking about • abstract terms, ideas, and relations.Opportunities to search and formulate abstract relations (write an equation, ex-• press the relation between the variables and the constants in a simple proportion, construct the formula for the calculation of properties of certain objects, analyze various mathematical, physical, chemical, economical, and other formulas, draw the structure of the problem situation, alternative strategies of the problem solving, etc.), use them when considering and solving problems.4.b Activity level in the educational activities (15 points)

4. c
Constructiveness in learning activities (10 points) 4.d Intentionality in learning activities (10 points) 4.e Authenticity in learning activities (10 points) 4.f Taking into account the social nature in learning activities (10 points) 4.g Taking into account the requirements following from the behavioural learning theory (26 points) 4.h Taking into account the requirements following from the cognitive theory (20 points) 4.i Taking into account individual differences (general and academic intelligence) (15 points) 4.j Taking into account individual differences (learning and cognitive styles) (10 points) 4.k Taking into account individual differences (profile of multiple intelligences) (5 points) 4.l Taking into account individual differences (types of preferred learning motivation) (5 points) Application runs under a secure protocol (e.g.HTTPS)  (0-4 points) [PC, MOB] In case the application stores the data in files, they are adequately  Used picture and film material is pertinent, easily identifiable, and comprehensible,  corresponding to the text information (0-4 points) Quality of the picture and film material used in the software is high, with clear  colours, developing aesthetic feeling in users (0-2 points)2.a.4Software content meets the social correctness requirements, is processed in  an objective manner and in compliance with the respect for human rights and freedoms Software enables users to monitor accurately their operations inside the software  structure, repeat them, or it can remember a direct path to it (saves it in the memory) (0-2 points) Software operation tools include the functions of fast search, return to the start,and step back (0-2 points)Software does not respond to a user's steps related to the lack of attendance,  accidental motion, or which are aimed at damaging its functionality (0 -2 points) Software operation does not hinder the work with other program applications and  Software contains majority of well-known information formats -photographs,  pictures, animations, audio and video recordings, ... (0-4 points) All the pieces of information are mutually interconnected by means of hypertext  links (0-2 points) Software works in the environment of independently controlled floating windows  that can be individually positioned, activated and hidden (0-2 points) Software thoroughly, cartographically, and accurately describes spatial relations 

)
Thanks to the diversity and variety of the content, the software enables a teacher  to use a wide range of teaching methods and procedures (0-4 points) Software is open and assumes the provided information database will be  subsequently used in the learning process and does not hinder that (0-4 points) Software can be used in all stages (sections) of a lesson  (0-4 points) Software contains the functions enabling (simplifying) the printing, storage,  sharing, or export of the presented information (0-6 points) 3.a.2Information texts are linguistically correct, stylistically adequate, used vocabulary  corresponds to the age of users in the target group (0-2 points) Information texts are written in an interesting and comprehensible manner,  sentence length and text segmentation into paragraphs corresponds to the age of users in the target group (0-2 points) Hypertext links refer to other information sources and the related facts are  highlighted in the text block and easy to distinguish (Software contains interactive educational tasks (tests) and continuously or  immediately evaluates the success rate of answers (0-6 points) Software contains interactive educational games, motivation and entertainment  activities related to the given topic (0-6 points) Software contains themes for educational projects, problem tasks, work sheets  in a digital form, and a handbook on how to efficiently work with the software at the lessons (0-8 points) (  Testing software ) Software monitors and evaluates in graphs and tables the continual success rate in the testing of several users (0-4 points) (  Testing software ) Software provides the success chart of individuals within the group, informs about their progress in time and in certain testing stages (0-4 points) (  Testing software) Individual testing items (questions, tasks) in the interactive tests are explicit and use various multimedia formats (0-6 points) (  Testing software) Software enables a teacher to lead, control, and organise Software enables to choose tasks from the database and produce specific tests based on a user's requirements (0-2 points) (Testing software) Software enables printing out of all the produced tests and administer them in a conventional manner (in writing) (0-2 points) 20 Tasks focused on learning new terms and their mutual relations based on analyses, comparisons,  classification, and abstraction of term properties (0-7 points) Simple attempts or experiments in the virtual environment containing the demonstrative  components (0-4 points) 4.a.2 (ISCED 2 and ISCED 3a) Opportunities to consider possible, hypothetical, and real situations and possibilities and also  the consequences of their update (solution of tasks with unreal input data) (0-8 points) Complex tasks requiring systematic solution procedures, search through the possibilities,  creation of strategies, probability-based thinking ... (0-5 points) Mediation of information requiring understanding, interpreting of, and thinking about abstract  terms, ideas, and relations (0-4 points) Opportunities to search and formulate abstract relations (write an equation, express the relation  between the variables and the constants in a simple proportion, construct the formula for the calculation of properties of certain objects, analyze various mathematical, physical, chemical, economical, and other formulas, draw the structure of the problem situation, alternative strategies of the problem solving, etc.), use them when considering and solving the problem (0-7 points) 24


of the curriculum content (0-2 points) Learners' possibility to intensify their understanding of the curriculum content by active  cognitive activities with regard to their own needs and interests (applying the gained knowledge and skills in other cognitive and practical contexts) They can choose tasks and problems, management of which relates to the learning objectives  selected by them and takes into account their aspiration level corresponding to the task difficulty (0-3 points) Learners can express and satisfy their cognitive needs and interests activated in them during the  study (extend, intensify their knowledge and skills in other areas or activities) (0-4 points) 104.e Taking into account the authenticity in learning activitiesLearners acquire the curriculum content in sufficiently elaborated connections with reality and  its practical application (0-3 points) Learners can solve tasks and problems in real and practical contexts  (0-4 points) Learners acquire not only the learning contents, procedures, but also their sources, either by  their own exploration or mediated via references to work of scientists or explorers (0-3 points) 104.fTaking into account the social nature in learning activitiesPossibility for learners to communicate, cooperate in the process of acquisition of the mediated  meaningful information (0-5 points) Possibility for learners to cooperate in solving tasks, problems, or projects, including learning Taking into account the requirements following from the behavioural learning theory Providing the continuous (immediate) and final feedback that informs the learners about the  correctness or faultiness of their solutions (0-3 points) The feedback includes instructions informing the learners on the subsequent procedure and are  of the motivation value for them (0-2 points) The program enables the learners make corrections and evaluate  (0-2 points)


(0-2 points) Text (verbal) information and conversational information should be interconnected, as for time  and space, with the picture information (0-2 points) Possibility for learners to enable and disable text or speech accompanying the pictures,  animations, etc. (0-2 points) In the digital learning material (program) there should not be any redundant accompanying  information, from the learning point of view (0-2 points) Respecting the perception principles (clear detachment of a figure -information of the  substantial importance from the background -the accompanying information) using a shape, colour, size, intensity of stimuli, by means of schemes etc. (0-2 points) Presentations containing picture information should include verbal information provided in  form of a speech and not a written text (0-2 points) Multimedia presentations in which visual materials are accompanied with speech, should not  contain text legends or explanations (0-2 points) Ensure balance and functionality of static and dynamic elements (e.g.animations should be  included only if they are functional -i.e. the objective is to provide demonstrative information of a procedural nature that contributes to the meaningful learning and not to distract attention towards efficient information, but of minor importance) (0-2 points) Provide learners with the preliminary information on the content and form of a multimedia  presentation and its components as the precondition for production of simple schemes of acquired terms and concepts (0-2 points) Enable learners to select verbal or visual information in the multimedia presentation (0-2 points) 26Enable learners to select the presentation speed, interrupt the presentation, or repeat any of its  components (0-2 points) Enable learners to select the content's or a task's difficulty level  (0-2 points) Enable learners to select the methods and time of topic revision  (0-2 points)4.iTaking into account individual differences in the level of general, academic intelligencePossibility to set several degrees of the curriculum content difficulty, reflecting various levels  of cognitive functions of learners (0-8 points) Possibility to set the working speed, working time, content scope, and tasks for individual Taking into account individual differences in learning and cognitive styles of learners Possibility to select visual or verbal modality of the provided information,When learning from the multimedia, enable the preference of visual or verbal modality of  the information (or the possibility of more or less intensive highlighting or suppression of the information of certain modality) (0-2 points) Representation of the sequence, hierarchy, structure of learning components and procedures  that facilitate structuring of the curriculum content and learning among learners with analytical cognitive style (0-2 points) Interconnection of learning components, curriculum segments, and concepts by means of  hypertext links, facilitating thus the formation of more complex cognitive representations of the curriculum content among learners with holistic learning style (0-2 points) Combination of procedures based on the programmed learning (learning material is structured  into smaller segments gradually presented to a learner) with simulations, modelling, and experiments in the virtual environment (learners can use the experience-based learning, adjust parameters of graphical representation of a process or a phenomenon, and monitor the consequences of their own manipulations, test the hypotheses, make experiments, learn by exploration) (0-2 points) Possibility to select a working speed, working time, content scope, type and difficulty level for sheets, handbooks on how to work with its digital content properly (in an appropriate and suitable manner) and use it reasonably in the education process.

•
Application does not require the Internet connection and its functionality is not • conditioned by a high speed of the Internet connection (transmission efficiency).Launch is smooth, fast, standard licence conditions are confirmed only at the first • launch.
Software is not a web application • Application does not require installation of any additional software (except for • the application's installation file).Installation runs smoothly, instructions for users are comprehensible, clear and • continuously inform users about the installation progress for individual segment.Application can be shared in a network (Intranet), or there is a network version • thereof.Application is launched directly from the medium, not requiring the assigned • administrator rights.Application is launched directly from the medium and the launch is not condi-• tioned by entering any special code with each new use, or access to the HW key.Detailed definitions were determined also for the groups of criteria evaluating the • technical, technological, and user attributes in the following areas: 1.b Stability (10 points) 1.c Support (12 points) 1.d Hardware (6 points) 1.e Operating system (5 points) 1.f Used technologies (5 points) 1.g Safety (4 points)

•
Software contains the functions enabling (simplifying) the printing, storage, shar-• ing, or export of the presented information.Information texts are linguistically correct, stylistically adequate, used vocabulary • corresponds to the age of users in the target group.Information texts are written in an interesting and comprehensible manner, sen-• tence length and text segmentation into paragraphs corresponds to the age of users in the target group.

Comprehensive Evaluation of Electronic Learning Tools and Educational Software (CEELTES) Area -Technical and technological characteristics
If a digital learning aid meets the criteria in the given sub-area, it is assigned the maximum number of points shown in brackets!In case it meets the criteria only partially, the score is reduced accordingly.If the evaluator is not able to assess objectively the applicability or inapplicability of any of the determined criteria, such criterion is skipped and the maximum number of points shown in the brackets next to the criterion will not be included in the overall score.) [WEB] Software is a web application (13 points)Application does not require installation of special plugins intended for specific