Design and Large-scale Evaluation of Educational Games for Teaching Sorting Algorithms

. The teaching of sorting algorithms is an essential topic in undergraduate computing courses. Typically the courses are taught through traditional lectures and exercises involving the implementation of the algorithms. As an alternative, this article presents the design and evaluation of three educational games for teaching Quicksort and Heapsort. The games have been evaluated in a series of case studies, including 23 applications of the games in data structures courses at the Federal University of Santa Catarina with the participation of a total of 371 students. The results provide a first indication that such educational games can contribute positively to the learning outcome on teaching sorting algorithms, supporting the students to achieve learning on higher levels as well as to increase the students’ motivation on this topic. The social interaction the games promote allows the students to cooperate or compete while playing, making learning more fun.


Introduction
Sorting algorithms, such as Quicksort and Heapsort, are essential topics in undergraduate computing courses (ACM/IEEE-CS, 2013; Poulova and Klimova, 2015;Chen and Fu, 2010).Following curriculum guidelines, it is expected that students learn competences related to these algorithms in data structures courses on the cognitive levels of comprehension and application (ACM/ IEEE-CS, 2013;Bloom, 1956).Typically these competences are taught through traditional lectures (ACM/IEEE-CS, 2013; Denton et al., 2005).This method is appropriate to present abstract concepts and factual information to a large group of students (Choi and Hannafin, 1995;Freeman et al., 2014).Often this method is complemented by exercises focusing on the implementation of these algorithms (Ernst et al., 2009).However, the use of other strategies such as Game-based Learning (GBL), can bring benefits that improve the learning by the application of this knowledge in a motivating and fun way (Poulova and Klimova, 2015;Hakulinen, 2012;Hakulinen, 2011;Freeman et al., 2014).
Game-based learning focuses on the application of educational games that are designed to balance learning to a particular competence with gameplay (Qian and Clark, 2016).They are currently adopted in computing education in diverse areas, such as software engineering, programming or security (Battistella and Gresse von Wangenheim, 2016;Boyle et al., 2016).Yet, so far they are not much used for teaching data structures knowledge.The few games that exist in this context focus mainly on teaching concepts related to the analysis of the complexity and/or other characteristics of the algorithms (e.g.Sorting Game and Sorting Casino (Hakulinen, 2011), DSAsketch (Hakulinen, 2012), Critical Mass (Lawrence, 2004), Active Learning (Budd, 2006), GBL for Data Structures (Tan and Seng, 2010)).Another shortcoming is the lack of sound empirical evidence of the benefits of these games in data structures courses.As currently most of these educational games were evaluated only in an ad-hoc manner and/or with very small samples (typically limited to the size of one class), the validity and generalizability of the results remain questionable (Calderón and Ruiz, 2015;Battistella and Gresse von Wangenheim, 2016;Boyle et al., 2016).
In this context, this paper proposes and evaluates three educational games to complement the learning of the algorithms Quicksort and Heapsort.The results of this research are expected to provide effective instructional alternatives for teaching sorting algorithms.Furthermore, the results of the large-scale evaluation of the games provide a sound empirical base for the adoption of the games in the classroom as well as identifying strengths and weaknesses of such games as a basis for improvement.

Sorting Algorithms
The teaching of sorting algorithms is typically part of the core content of data structures courses of undergraduate computing courses (ACM/IEEE-CS, 2013).As part of the data structures course, students need to learn to understand simple numerical algorithms, calculating the average value of data in a list , search the minimum and maximum values in a list, sequential or binary search algorithms, analyze the complexity of algorithms, create table-hash and perform basic operations in search-trees such as insertion, deletion, alteration (ACM/IEEE-CS, 2013).Usually, it is required that students know to implement stacks, queues, algorithms of quadratic complexity (Selectionsort, Insertionsort) and quasilinear algorithms (Quicksort, Heapsort, Mergesort) (ACM/IEEE-CS, 2013; Ernst et al., 2009).At the undergraduate level, it is expected that the students acquire competences on the cognitive levels of comprehension and application, according to the Bloom's taxonomy (1956).At the comprehension level it is expected that the students comprehend the design principles and the structure of the algorithm.At the application level, students are expected to be able to, step by step, apply the sorting algorithms to a set of random numbers without consultation of the written algorithm.The authors selected the Quicksort and Heapsort algorithms from the contents of an undergraduate data structures course for the production of educational games.In previous years, these algorithms were taught only through traditional lectures, as depicted in Fig. 2. Thus, the games have been created to allow students instrumentalise the knowledge they acquired during their study of the algorithms.
Quicksort is one of the most popular sorting algorithms in literature, being based on the divide and conquer algorithm design technique (Sane and Deshpande, 2006).There are several versions seeking to improve its operation (Munro and Ji, 1996;Stone, 1978;Prifti, 2015).The algorithm was proposed by Hoare in 1959, but is still taught in computing courses (Reilly, 2003;ACM / IEEE-CS, 2013).Quicksort starts by choosing an element of a previously unsorted array, which is called the pivot.Then, the algorithm compares the values of the array with the pivot.Values that are lower than the pivot are placed on the left of the pivot and, values that are larger are placed on the right of the pivot.This is the main operation of the Quicksort algorithm, called partition, where the array is subdivided into two disjoint subarrays.The algorithm repeats this step recursively with each subarray in order to complete the sorting of the array (Sane and Deshpande 2006).The complexity of the algorithm is quadratic (n²) in the worst case (when the array is sorted or inverse-sorted) and quasilinear (n log n) in the average case.It has been extensively analyzed (Munro and Ji, 1996;Stone, 1978;Prifti, 2015).
Heapsort is a sorting algorithm that uses a vision of an array as a left-balanced heaptree to perform the sorting of this array.It was developed by Williams and refined by Floyd in 1964 (Wegner and Teuhola, 1989).In order to sort the elements of the array, the algorithm performs a process called sifting-up.While searching the array, Heapsort looks locally at each father/children set, accordingly to the heap-tree vision, and places the highest value in the father and lowest in one of the children (Sane and Deshpande, 2006;Wegner and Teuhola, 1989;Bai et al., 2014).The logic behind this algorithm is to gradually move the larger elements to the beginning of the array (upper levels of the heap-tree), inducing a partial order where, for each subtree, each individual node is always larger as its descendants, without ordering the whole heap.When the largest value of the array is inserted into the root, the value of the first position should be replaced with the value of the last position.Whenever a value is moved to the end of the array, the array size is decremented and sifts-up the array again to ensure that the there is no father larger than its children (Sane and Deshpande 2006;Wegner and Teuhola 1989;Bai et al., 2014).The Heapsort algorithm is very stable and shows quasilinear complexity (n log n) in all cases.

Educational Games
A game can be defined as "any competition (game) between opponents (players) operating under constraints (rules) for a goal (win or profit)" (Abt, 2002).In this context, educational games are specifically designed to teach a particular competence (Qian and Clark, 2016;Abt, 2002;Dempsey, 1996).Different types of games can be used for education, including digital or non-digital ones.Digital (electronic) games involve interaction with an electronic device (e.g., a computer, console, tablet or smartphone).Nondigital games are manual games such as board games, cards or using simply pens and paper or other props (Connolly et al., 2012;Caulfield, Veal, and Maj, 2011).Games can be played in different modes of interaction, for example as single-player or multiplayer (Schell, 2008;Fullerton, 2008).
In an educational context, games can be adopted in order to focus on different levels of learning, ranging in undergraduate courses typically from the level of remembering (exhibit memory of previously learned materials by recalling facts and concepts), comprehension (demonstrate understanding of facts and ideas) to the level of application (solve problems to new situations by applying acquired knowledge) based on Bloom's Taxonomy (1956) (ACM/IEEE-CS, 2005).

Adventure
A game in which the player assumes the role of the protagonist in an interactive story driven by exploration and the solution of challenges/quests (such as decoding messages, finding and using items) in order for the narrative to progress.

Guessing
A game in which the object is to identify some kind of information, such as a word, from other player's drawings or mimicking.

Puzzle
A game that involves controlling game characters using a set of commands and executing them, to navigate through a maze or to arrange objects in a correct and efficient way to achieve the game goal.

Quiz
A game where the player is presented with trivia questions and must select or give the right answer.

Racing
A game in which the player commands a vehicle or participates in a race trying to move faster than an opponent to reach a specified goal or beat a specified time.
Roll-and-move Board games in which a player's token or tokens are moved based on results shown on a die or dice.

Role-playing game (RPG)
A game where the player controls the actions of a protagonist as this character lives immersed in a fictional world.A role-playing game can be seen as such when player-controlled characters become stronger ("levels up") because of the player's actions (usually depending on experience points received), rather than being upgraded automatically as dictated by the storyline.

Simulation
A game designed to put the player in control of a certain environment or activity being mimicked as realistic as possible.

Strategy
A game that focuses on gameplay requiring careful and skillful thinking and tactics in order to achieve victory.

Teaching Sorting Algorithms with Educational Games
Considering the current tendency to adopt educational games for computing education, they are still not widely used for teaching of sorting algorithms (Battistella and Gresse von Wangenheim, 2016a;Boyle et al., 2016).Based on a literature review, we only identified 7 educational games focusing on the teaching of sorting algorithms in higher computing education as presented in Table 2. Game that covers a wide range of data structures and algorithms concepts in order to recap the course material.Players are divided into two teams.The main idea of the game is to draw concepts related to data structures and algorithms to the whiteboard, while other team members are trying to guess the concept in the picture.The winner of the game is the team with the most points after all players have had a turn to draw.In this game, which resembles the card game casino, the player uses cards for defining characteristics of a sorting algorithm, for example, setting its complexity.On his/her turn, the player can capture one or more cards from the table by using one of her/his hand cards.With an algorithm card, the player can capture all special cards that are valid for that algorithm.A player can put all the captured cards to his victory stack.The game ends when either the algorithm stack or special card stack is empty.The winner is the player with most cards in his/her victory stack.

Non-digital Multiplayer
Action Complexity of sorting algorithms Comprehension; Remembering (Hakulinen, 2011) Sorting Game Name Image

DSAsketch
Lightest and Heaviest

Sorting Casino
Sorting Game

Sortko
This game uses the same cards as used in the game Sporting Casino.
The basic difference is that in this game are used special cards on complex algorithms.In the beginning, the dealer deals 3 algorithm cards and 2 special cards to each player.The actual game round consists of two phases: special card phase and algorithm phase.
During the special card phase, each player can place one special card to the Designed using an interactive graphical user interface with an option of choosing a sort (e.g.Bubblesort, Shellsort, Quicksort, Insertionsort) and generates a random sequence of numbers to be sorted by applying a sequence of specific interaction sorting gestures.During the sorting process, the system informs students about their success in performing so called "switches" and provides them with textual help messages which facilitate further progress.
As a result of one game, a student gets rewarded with points that are then summarized on a joint public list.Win the game, the player that performs the ordination correctly of values.The game was designed to be played on Android phones.

Single-player Simula-tion
Sorting algorithms Bubblesort, Shellsort, Quicksort, Insertionsort.Application; Comprehension; Remembering (Boticki, Barisic, Martin and Drljevic, 2012) This shows that there are currently only a few educational games for teaching sorting algorithms available, including digital as well as non-digital games.The non-digital games rather focus on lower learning levels (remembering and comprehension) in contrast to some of the digital games that also aim at learning on the application level.And, although, several of the games aim at teaching the application of the sorting algorithms, their benefits remain questionable due to a lack of (rigorous) evaluations with sample sizes large enough to obtain significant results.We also observed a lack of information on how the games have been developed and their characterization with respect to educational aspects.

Research methodology
This article aims at the conduction of an exploratory research related to games to teach Quicksort and Heapsort sorting algorithms.A case study was performed to understand the phenomena observed by the application of the educational games (Fig. 1).
The case study is performed according to the procedure proposed by Yin ( 2013) and Wohlin et al. (2012): Study definition.Using the approach Goal Question Metric (GQM) (Basili et al., 1994) the purpose of the study is defined and systematically decomposed into analysis questions and metrics based on the MEEGA model (Gresse von Wangenheim et al., 2012;Savi et al., 2011).MEEGA evaluates the quality of educational games with respect to their impact on motivation, user experience and learning.It is currently one of the most commonly used models for the evaluation of educational games (Calderón and Ruiz, 2015;Petri and Gresse von Wangenheim, 2016).Based on the analysis questions the standardized MEEGA questionnaire (Savi et al., 2011) is used.

Study execution.
The instructional design of the games was done following the ADDIE model (Branch, 2009).Initially, a context analysis was carried out analyzing the target audience and environment, as well as, identifying learning requirements.According to the context analysis, learning objectives and an instructional strategy were defined, including the content, sequence and teaching methods to be adopted.In accordance with the defined instructional strategy, the games were developed taking into consideration fundamental aspects of game design (Fullerton, 2008;Schell, 2008).The games were applied in several classes in undergraduate computing courses.The evaluation was carried out by a series of case studies with a one-shot post-test only design, in which the educational games were applied and afterward data was collected from the students using the MEEGA questionnaire.
Analysis and interpretation.The collected data was analyzed in accordance to the defined analysis questions, using descriptive statistics as indicated by the model MEEGA (Savi et al., 2011).The results were interpreted taking also into consideration further observations made during the applications.

Context Analysis
With the objective to improve the teaching of sorting algorithms in undergraduate courses, the target audience is students in the first semesters of computing courses (such as Computer Science and Information Systems).Typically these students already have acquired a basic knowledge of object-oriented programming and repetition and decision structures.Regarding data structures, students usually already have knowledge on sorting algorithms of simple quadratic complexity, such as Bubblesort.
In general, computing students are mostly young (19 to 30 years) males.Commonly they play digital games, mainly action and adventure games and RPG (Role-Playing Games) for entertainment (Battistella and Gresse von Wangenheim, 2016b).With respect to instructional strategies, the students prefer classes that promote active participation and social interaction with their classmates (Budd, 2006;Qian and Clark, 2016).
In the context of Brazilian public universities, class sizes typical range from 20 to 35 students.Classes are either held in classrooms equipped with a whiteboard and data projector, or computer laboratories.These laboratories typically provide 15-20 computers with Internet access using Windows and/or Linux operating systems (Battistella and Gresse von Wangenheim, 2016b).
Specifically in undergraduate courses at the Department of Informatics and Statistics at the Federal University of Santa Catarina (UFSC), the data structures course is a onesemester course with 108 class hours.On average, 12 hours are spent on teaching sorting algorithms.The teaching of sorting algorithms is divided into two parts in accordance to their complexity.First, quadratic sorting algorithms are taught (such as Insertionsort, Selectionsort, Bubblesort) and after that, the quasilinear algorithms (typically Quicksort, Shellsort and Heapsort) are taught.The learning objective of this instructional unit is that at the end of the institutional unit on sorting algorithms with quasilinear complexity, students should be able to understand and apply the sorting algorithms.
Aiming at improving the course, we chose sorting algorithms because this subject was being addressed in a rather abstract way, through traditional lectures based upon structural analyses of the algorithms, where we observed conceptual comprehension difficulties among the students.We specially selected Quicksort, because it is a well-known sorting algorithm that is a standard in data structures course and should be well understood by the students.Heapsort was selected as its approach differs conceptually from most common sorting algorithms and we observed that it poses a learning challenge to the students.

Design of the Educational Games
Taking into consideration our observations of the major difficulties experienced by students in understanding sorting algorithms with quasilinear complexity and also taking into account the characteristics of the context and the learning objective, we identified the opportunity to improve the instructional strategy.We introduced educational games where the student plays an active role.The student enacts the algorithms through the use of a physical board game that is used, together with poker chips, as a memory model for the manual simulation of the Quicksort e Heapsort algorithms.This contributes to learning mainly in the levels of understanding and application with respect to the conceptual execution of the algorithm.At this point this does not include any learning objective with respect to the implementation of these algorithms.These contents are presented through traditional lectures, followed by a video that shows a simulation of the usage the board on each game (http://www.incod.ufsc.br/novaaula/heapsort-simulation/and http://www.incod.ufsc.br/novaaula/quicksort-simulation/)and then the sorting games are applied, as shown in Fig. 3.In addition, students also use the digital game for Heapsort algorithm.To promote learning on the levels of comprehension and application, simulation games were developed, allowing the students to execute the sorting algorithms step by step.Initially, both board games were created and, then, in order to provide also an alternative beyond the classroom, a digital version of the heapsort game was developed.

Quicksort Board Game
The Quicksort Board Game is a board game that can be used to represent the memory model of a sequence of recursive calls of the Quicksort algorithm.It allows simulating the execution of the individual steps of the Quicksort algorithm (Fig. 4).The game was designed to be played in the classroom in groups of 4 to 6 students with a duration of about 60 minutes.The game uses a board representing the memory area of the values to be sorted, and the contents of the control variables used by the algorithm in the form of an inverted stack containing "photographs" of the subsequent recursive calls to the algorithm.
To play the game, numbered chips are used with values between 0 and 100 arranged in a random sequence on the first row of the array stack of the board.The numbers are inserted on the board randomly, so when starting the game the complexity of the sorting falls in the average case, quasilinear (n log n).Players must use the pointer called "pivot" to sort the values.During each iteration, the array is divided and players must create two new sub-arrays and stack them in a row below, always choosing a pivot for each new sub-array.In the end, the sub-arrays are combined resulting in the sorted array.The group that finishes the sorting first with all values correctly sorted, wins the game.A teaching assistant accompanies the students while they play to ensure that they are correctly following the steps of the Quicksort algorithm.A printable model of the board of the game is available (in Brazilian Portuguese) at http://www.inf.ufsc.br/~aldo.vw/sorting/tabuleiro-sorting-A1.pdf.In the game, student groups should sort numbers by following the steps of the Quicksort algorithm by the students as a group.Within each group the students work in a collaborative way all in the same role as an executor of the algorithm.During the game, the students in each group discuss among themselves the next steps to be taken to sort the data and then execute the step.

Heapsort Board and Digital Game
The Heapsort Board Game is a board game simulating the execution of the steps of the Heapsort algorithm (Fig. 5).The game is also designed to be played in groups of 4 to 6 students in the classroom with an approximate duration of 60 minutes.In the game, students need to order 21 values (random numbers from 0-100) in an array, accordingly to the Heapsort algorithm.The game employs the same game board of our Quicksort game, pointers are, however, positioned in different fields and control variables have different meanings.In this case, the memory model represented by the game board is used to simulate the heap tree and the control variables used by the algorithm to traverse this heap during the sifting-up process.In order to play, players must place the numbered chips listed in the array and should position special colored poker chips in the positions pointed by the father, child and index variables of the algorithm.Then, the players must decrement the index variable until reaching 0, making sure that, at every iteration, every parent node in the heap tree contains a higher value than its children nodes.Children are found from the formula (index * 2) and (index * 2 + 1), where the index is the position of the father node in the heap.When a heap is finally built (each parent is larger than its children), the player must move the root (first array element) to the last position.The last unsorted element will be moved to the root, and a sifting-up is performed, to guarantee that a new heap is generated.The game ends when the unsorted array has size 1.The group that finishes the sorting first, wins the game.As in the Quicksort the game, a teaching assitant accompanies the students while they play to ensure that they are correctly following the steps of Heapsort algorithm.A printable model of the board of the game is also available (in Brazilian Portuguese) at http://www.inf.ufsc.br/~aldo.vw/sorting/tabuleiro-sorting-A1.pdf.In the same ways as in the Quicksort game, the students work together in groups, with all students having the same function of sorting the numbers following the steps of the Heapsort algorithm.The Heapsort Digital Game is a digital version inspired by the Heapsort Board Game (Fig. 6).As the non-digital version, the game seeks to simulate the execution of the Heapsort algorithm, requiring the student to carry out the steps that the algorithm would use to sort the data.The game is played individually so that the student has to sort the values of the array "Array values" using "Father" and "Child" to sort values.To win the game, the student must have ordered all the values of the array, following the steps of Heapsort.
To play the game, students should click the "Start" button.Thus, should start ordering the values, using the "Replace values" button to change the value in between the pointers "Father" and "Child".The "Remove root "button is used to switch the first element of the array, with the last element.This button is used when the highest value of the array is at the root.When the player clicks the button, this element is moved to the end of the array.The game also provides a video explaining the operation of the algorithm and the rules of the game.Students learn these rules beforehand during the traditional lectures, but the game also presents rules of the game.
The game has been developed using a client-server architecture to be played online via the browser.The game was developed using the HTML5 language, JavaScript and PHP, and is available online at http://www.incod.ufsc.br/sortia.

Application of the Educational Games
The games have been applied in data structures courses at undergraduate computing courses (Computer Science and Information Systems) at the Federal University of Santa Catarina (UFSC)/Brazil.Since 2012, the games have been applied in 23 courses with a total participation of 371 students (Table 3).The applications of the games were conducted by the teaching assistant supervised by the lecturer.
According to the defined instructional strategy (section 3) the games have been applied after the respective traditional lectures.In case of the non-digital game, the students were divided into groups of 4 to 6 students.Each group received a board and tokens to mark the pointers of numbered tokens (0 to 100).The mechanics and the rules were explained by the teaching assitant and the students then played the game.Fig. 7 illustrates scenes from the application of the Quicksort Board Game and Heapsort Board Game.A video showing some student groups playing the game are available here: http://www.incod.ufsc.br/novaaula/sorting-algorithm-board-game-video/.In general, students played each game more than once, first to understand the operation of the game and, then again in order to play the game more effectively.The application of both nondigital games took place in classrooms.
The applications of the Heapsort Digital Game took place in computer labs.Even though the game was designed to be played individually on a computer, some students played in pairs due to the limited number of computers available, yet also allowing students to help each other.In the beginning, the teaching assitant explained the features and rules of the game.
At the end of each application, students evaluated the games by answering the MEEGA questionnaire (Savi et al., 2011).An exception has taken place in the second semester of 2013, when students played both the Heapsort Board Game and Heapsort Digital Game and, thus, a single questionnaire has been used to evaluate both games together.

Definition
The overall objective is to evaluate the quality of the developed games.Due to the lack of a commonly accepted definition for the quality of educational games, we define a high quality game as one that clearly presents the educational objectives, motivates students and promotes learning curriculum content in a fun, enjoyable and challenging way (Savi and Ulbricht, 2008).Thus, the objective of the evaluation is to analyze the quality (including motivation, user experience and their contribution to learning) of the games Quicksort Board Game, Board Game and Heapsort Digital Game, in the context of data structures courses of undergraduate computing courses from the student's point of view.
Based on this objective, we define the following analysis questions: AQ1. Do the games motivate students?AQ2.Do the games provide a positive user experience?AQ3.Do the games contribute to learning?AQ4.What are differences in the quality of the Quicksort Board Game, Heapsort Board Game and Heapsort Digital Game in relation to motivation, user experience, and learning?
The research hypothesis is that the games contribute to the achievement of the learning objectives of the instructional unit, motivate students and promote a pleasant user experience.Based on the MEEGA model (Savi et al., 2011) the following dimensions and measurement items are defined as shown in Fig. 8.
For data collection we use the standardized MEEGA questionnaire (Gresse von Wangenheim et al., 2012;Savi et al., 2011) addressing the measurement items as shown in Fig. 8 on a 5-point Likert scale ranging from strongly agree (2) to strongly disagree (-2).In addition, data is collected on a self-assessment of the cognitive level of students before and after the application of the game in relation to the specific learning objectives of every game.The achieved learning level with respect to each learning objective is evaluated by students in relation to their knowledge before and after the game application on an interval scale ranging from 0 to 5. Fig. 8. Overview of dimensions and measurement items based on the MEEGA model (Savi et al., 2011).

Analysis and Results
In order to obtain greater precision and statistical relevance, increasing the sample size, the data collected from the 23 applications were analyzed cumulatively.The grouping of the data was possible due to the similarity and standardization of the data collected.
In this series of case studies, the data are similar in terms of definition (evaluating an educational game in relation to motivation, user experience and learning), research design (case study) and context (undergraduate computing courses).All data collected are standardized with respect to the measures (Fig. 7), the data collection instrument using the MEEGA questionnaire (Savi et al., 2011), and the response format (using a 5-point Likert scale and interval scale).
In accordance with the MEEGA model, the collected data has been analyzed using descriptive statistics with respect to each of the defined analysis questions.

AQ1. Do the games motivate students?
In general, students recognized positively the motivation provided by the games, as shown in Fig. 9.The majority of the students was satisfied as the games allowed them to apply the knowledge taught beforehand, providing an alternative way to learn the application of the sorting algorithms.A large majority of the students felt confident that they were learning as they played the games.We also observed that students that had obtained already a better understanding helped their colleagues in a way of peer instruction.Most students considered the contents of the games relevant within the course context.The large majority of the students also considered the design of games attractive, contributing positively to the increase of their interests.AQ2.Do the games provide a positive user experience?
In general, the students evaluated the dimensions competence, fun and social interaction very positively (Fig. 10).Most students are confident that they are learning in an efficient manner with the games.In general, the students had fun while playing, creating a more relaxed setting during class.One of the most positively evaluated dimensions is the social interaction promoted by the games, whether competitively or collaboratively.Often students helped their colleagues contributing positively to their learning, whereas on the other hand, the competition promoted by the games between groups kept the students focused on the learning task.
On the other side, the games were evaluated moderately with respect to immersion.As immersion is important in order to keep the students focused, it would be important to improve this aspect of the games, for example by stimulating further the competition or create engaging narratives.

AQ3. Do the games contribute to learning?
In general, the games seem to contribute positively to the student's learning as shown in Fig. 11.The students affirm that the games are effective instructional strategies for learning also in comparison to other instructional strategies used in the course.The stu- dents assert that the games lead them to learn on cognitive levels of knowledge, understanding, and application, as presented in Fig. 12.
AQ4.What are differences in the quality of the Quicksort Board Game, Heapsort Board Game and Heapsort Digital Game in relation to motivation, user experience and learning?
Comparing the evaluation of motivation provided by each game, we can observe that the results were positive in relation to satisfaction, confidence, relevance and attention with respect to all three games (Fig. 13).It is noteworthy, that there have been no evaluations with negative or neutral median for any of the games.In general, the evaluations among the three games are rather consistent.However, the digital game seems to be slightly better evaluated with respect to the ease to play the game as well as the attractiveness of its design.In the digital game, the information the names of pointers ("Father" or "Child") seems to be clearer that board games.Thus, allowing students to understand more easily the features of the game.
Comparing the evaluation of the games with respect to user experience a similar positive result can be observed as presented in Fig. 14.Exceptions are items related to immersion as well as the students' disappointment when the class ended (neutral me-    dians).With respect to user experience, the digital game has only been slightly better evaluated than the non-digital games with respect to being recommended to colleagues.On the other hand, the Quicksort Board Game has been highlighted with respect to its very positive evaluation of social interaction.
Comparing the evaluation of the contribution to the learning of each of the games, we can observe a complete consistent result across the three games.

Discussion
The results obtained from the evaluation of the games are quite positive.Students noted that the games helped to reinforce the learning of these sorting algorithms in an engaging way, as well as motivating them.Overall, the quality dimensions of the games were highly rated, with medians varying between 1 and 2. In general, students actively participated in the games applications, playing the games with ease and having fun.Especially, the social interaction promoted by the games was commented very positively by students.Even in case of the digital single-player game, social interaction was evaluated positively as students played the game in pairs (two students per computer) and the promotion of competition through the ranking between students.
Students considered the games relevant within the context of the data structures course and being connected to the knowledge they already acquired.They also expressed that they consider the games important to their personal interests.We observed that this instructional strategy kept the attention of the students focused, especially in the beginning, when they concentrated on understanding the application of the sorting algorithm.When compared to other instructional strategies, games seem to promote competition and social interaction in an intrinsic manner contributing to keep the students easily engaged in the learning task.Another aspect highlighted by the students was the fun they had while learning sorting algorithms creating at the same time a more relaxed learning environment.
On the other side, one of the dimensions less positively evaluated was immersion.Thus, an improvement opportunity can be the creation of new challenges that lead students to greater competitiveness and rewards for the winners and/or the integration of an engaging narrative.Comparing the three games, no significant differences have been observed neither in terms of the sorting algorithm nor the type of game (digital vs. non-digital games).
Based on student evaluations, we can observe that with the games students increased their knowledge on sorting algorithms contributing positively to the achievement of the learning objectives.The games were assessed positively as instructional strategies by the students also in comparison to other strategies used in the discipline.
Threats to Validity.The results obtained in this study need to be interpreted with caution, taking into account potential threats to their validity.One threat may be the research design adopted conducting a series of case studies (one-shot post-test only).Due to the lack of a pre-test measurement, learning achievements cannot be compared to the acquired knowledge before the application of the games.This threat is further compounded by the fact that there is no control group to compare the identified effects.However, in this research, we chose to conduct a case study rather than a formal experiment in order to minimize interruptions in the normal course flow.This may reduce the validity of the results obtained, yet due to practical reasons was the only possible way to perform the research.
Another threat to validity is related to the sample size, which may compromise the generalizability of the results.To minimize this problem, we conducted a large-scale series of case studies being conducted over 4.5 years, with the total participation of 371 students representing a much larger sample than other evaluations of similar games encountered in literature.
There may also be threats to construct validity.Due to practical limitations, running the study as part of a class, the results related to learning effects were obtained from self-assessments of the students.This type of assessment may not be enough to measure students' knowledge, however, there is evidence that self-assessments results provide reliable, valid and useful results (Topping, 2003;Falchikov and Boud, 1989).Considering the complexity of an assessment of educational games, the evaluation of the perception of students is a simple, fast and less invasive approach for feedback.
Another possible threat is that the quality aspects of the game are difficult to measure.This fact is mainly due to its high degree of subjectivity.To counteract this threat to validity, a standarized questionnaire has been used based on a commonly evaluation model (MEEGA) that has be evaluated with respect to its validity and reliability (Savi et al., 2011).

Conclusion
Motivated by the lack of educational games to teach the sorting algorithms Quicksort e Heapsort, we systematically developed three games Quicksort Board Game, Heapsort Board Game and Heapsort Digital Game in order to support the teaching of quasilinear sorting algorithms with logarithmic complexity.The games were applied in 23 data structures courses at the undergraduate computing course at the Federal University of Santa Catarina with a total participation of 371 students.Results indicate that the games allow students to achieve the learning objectives of the instructional unit as well as to motivate them and to provide a positive learner experience.Students considered the games relevant within the course context allowing to reinforce the learning of these sorting algorithms on the cognitive levels of comprehension and application.Furthermore, students recognized that they had fun by playing the games enjoying the social interaction through cooperation or competition among colleagues.In general, based on the positive evaluation of the students, the games were integrated permanently in the data structures course as a complementary instructional method to support the teaching of the algorithms Quicksort and Heapsort.
small boxes with different weights using a scale.Applying the algorithms students must choose between couples of boxes based on their weight to order them.with the Microsoft XNA Game Studio created for teachers to develop sorting algorithm games.Three existing games prototypes focus on binary search algorithm, singlylinked list and binary search tree.To create a new algorithm game, the developer needs to create a new algorithm game project.Then, the developer must set up the game properties, assets, graphics, items, classes and screens using the Properties, Assets, Graphics Items, classes, and Screens Game Editors respectively.After that, the developer needs to implement the required methods of the algorithm game, and adds any needed code.Finally, the developer needs to compile, debug and execute the created algorithm game.and sort them according to their values by using sorting algorithms (Bubblesort and Mergesort).The game features a medieval scenario based on the famous game World of Warcraft (RPG).The game supports single player and multiplayer modes.Both game modes support spectator mode, which can be used by any lecturer to act as a facilitator.To win the game, it is necessary to correctly order the boxes, from lowest to highest.Digital Single

Fig. 2 .
Fig. 2. Overview on instructional strategy of the data structures course.

Fig. 4 .
Fig. 4. Board Game for Quicksort Algorithm.Rows labelled limInf/limSup represent the stack of recursive calls to Quicksort and the corresponding values of the local variables containing each subarray lower and upper boundaries limInf and limSup.

Fig. 7 .
Fig. 7. Scenes of students playing the board games.

Fig. 12 .
Fig. 12. Auto-assessment of the achievement of learning objectives (total number of responses: 371).

Fig. 13 .
Fig. 13.Comparison of the medians of the evaluation of motivation (total number of responses: 371).

Fig. 14 .
Fig. 14.Comparison of the medians with respect to the evaluation of user experience (total number of responses: 371).

Fig. 15 .
Fig. 15.Comparison of the medians with respect to the evaluation of learning (total number of responses: 371).

Table 3
Overview on the applications of the games