Improving the Usability in a Video Game Through Continuous Usability Evaluations

Abstract

User experience in a software product - like a videogame - has a great impact on the success and acceptation of it. Players feel better when they can learn and understand easily the game’s goals and how to reach them. The purpose of this study is to show evidence that continuous user experience tests combined with software improvements can make that acceptation reached in a smarter way. These tasks are accomplished integrating usability techniques and Agile Methodology.

1 Introduction

The development of video games usually includes the following major activities: game design, development, testing and deployment. The development phase involves creating the necessary graphics and sounds as well as building the software that implements the video game [8]. Like any software, a video game must be created applying a development methodology but as a highly interactive element, such methodology should be a flexible process that allows usability activities to be introduced.

According [2], a videogame is an interactive software conceived “to make the player feel good when playing it”, unlike other interactive software that are made for accomplishing tasks in a specific context.

Usability is a software-product quality measure system, used to ensure that users reach objectives in an efficient, effectively and satisfactory way in a specific context [3]. This means that a videogame quality cannot be ensured using usability, it needs a broader concept like user experience.

This paper describes the experience of making a high-level acceptation videogame, combining a software methodology with continuous user experience test iterations. In further sections development methodology and testing instruments will be described, and finally results will be discussed.

2 Methodology

In this section we will describe Usability and the Agile Methodologies, as well as the integration of both techniques for the development of the tool that is part of the present study.

2.1 Usability

According to [3], the definition of usability is: the effectiveness, efficiency and satisfaction with which certain users reach the specified goals in particular environments.

In [4], Nielsen indicates that usability engineering is not a single activity that takes place before the launch of the product, on the other hand is a set of activities that take place throughout the product life-cycle management.

Nielsen presents the following steps as part of the life cycle model of usability engineering:

• 1. Know the user.

• 2. Competitive analysis.

• 3. Set usability goals.

• 4. Parallel design: Develop multiple versions of a parallel interface, compare the results and take the best aspects of each.

• 5. Participatory design: Involve the user as part of development and allows developers to have a group of users available to make queries.

• 6. Coordinated design of the entire user interface: Coordinate and centralize aspects of the interface to maintain consistency throughout the application.

• 7. Apply guides and perform heuristic analysis.

• 8. Prototyping.

• 9. Empirical evidence.

• 10. Iterative design: The application must be developed in test and correction cycles based on the results of the empirical tests.

• 11. Collect feedback from field use.

Rubin and Chisnell [5] defines user-centered design as techniques, processes, methods and procedures for designing usable products and systems as well as the philosophy that places the user at the center of this design process.

The proposed lifecycle by [9] as the user-centered development lifecycle is shown in the Fig. 1.

Fig. 1.

User-Centered Design Process Map [9]

Both documents [4, 9] propose common tasks such as:

• Knowledge of the user, necessary when defining requirements of any type of tool.

• Usability testing.

• Prototyping.

• Heuristic evaluation.

• Iterative design.

2.2 Agile Methodology

Agile methodologies are a group of software development methods that emphasize [2]:

• The collaboration between the programming team and the business experts.

• Communication face to face.

• Rapid delivery of value for the business.

• Self-organized teams.

• The quick response to change.

The most used agile methodologies are XP and Scrum [7]. The latter allows organizing medium teams so it was a matter of our interest.

Scrum [6] is an agile approach for product and service development. The work is done in sprints that are fixed periods of time of between 1 to 4 weeks during which a multidisciplinary team does all the work. At the end of each period the team must have built a potentially deliverable product.

In Scrum [6], user stories that an application must deploy are placed in a product backlog consisting of a prioritized list of features. The activities that are carried out in Scrum and that allow to advance the product backlog are:

• Product backlog grooming: new functionality of the product backlog is added or removed.

• Sprint planning: planning phase prior to each iteration.

• Sprint: an iteration during which the product is built.

• Daily scrum: 15-minute daily meeting of the team, during which the best way to move towards the solution is reviewed.

• Sprint execution: construction activity of the development team.

• Sprint review: inspects and adapts the product at the end of each iteration.

• Sprint retrospective: the construction process is reviewed and adapted at the end of each iteration (Fig. 2).

  Fig. 2.

  

  Scrum life cycle [6]

The roles in this methodology [6] allow to properly coordinate the team and are the following:

• Product Owner: The person responsible for product leadership and define what features should be built, in what order to build and prioritize.

• Scrum Master: A person who provides leadership to the team, acts as coach of the principles of Scrum, it helps solve blockers and the team achieve its best performance.

• Development Team: The self-organized team who is responsible for designing, building and testing the desired product.

2.3 Integrating Usability and an Agile Methodology

The proposed integration is based on a case study by Aguilar and Zapata [1], in which the XP design process is improved through the use of UCD tools. The difference for the present work is that when developing a video game, we are facing a more extensive application in development time and work equipment, so it was decided to use Scrum instead of XP as shown in Fig. 1. In this sense, “Interviews” will not be used because the requirements were obtained from the game design but “Personas” and “Scenarios” were used because they helped to represent different groups of users and their needs. This allowed bettering defining the game mechanics as the forms of interaction and the design of the interface.

“User Evaluations” will go hand in hand with development iterations and participants will be required to evaluate the degree to which the product meets predefined usability criteria (Fig. 3).

Fig. 3.

Adaptation of integration proposed [1]

3 Development Process

The objective of the project was to develop a videogame that motivates high school students to learn more about Mariano Melgar, who was an important poet during the Independence of Peru. The platform game is placed in a dream of the character and through the levels moves from Mariano Melgar’s love conflicts to his surrender for the freedom of his country. The Fig. 4 shows some stages of the game.

Fig. 4.

First level of the videogame

3.1 Personas and Scenarios

The group of high school students who are wanted to reach are adolescents between 13 and 16 years and designed the following possible profiles:

• Students with good performance: organized, curious and with great motivation for reading. When faced with a video game you are interested in deepening its content (story, characters and possible related stories).

• Students disinterested: unmotivated and dedicated to leisure activities, whose objective is only to approve the courses but not necessarily to learn. When faced with a video game he wants to be entertained and does not see it as a learning tool.

• Students gamer: has a great interest in video games but has a hard time seeing them as a learning tool. When faced with a video game, he expects to have high quality graphics and music and that the mechanics and history of the game are fun.

• Students not gamer: has little interest in video games but uses various technological tools such as social networks. When faced with a video game, it is expected to be easy to play.

3.2 Iterations

The video game has two levels: the first shows the user how to use the commands and meet the objectives; on the other hand the second level gives more information about the character and his role in the Independence of Peru. These levels were developed in four iterations where, after each one, user evaluations were performed.

3.3 Continuous Evaluations

The objective of the evaluations done was to obtain information about the user experience in the game. This experience includes the understanding of the main goal of the game, and learning the actions/movements made by the character.

People for the first three iterations were chosen randomly, only with the requirement that they can operate a computer, so their experience in previous games can be similar or different. It was expected that they learn themselves to discover how to achieve the main goal and the tasks need for that. Except for a unique user case, users chosen for one test didn’t participate in other tests. For the fourth iteration, it was proposed an age and gender balanced group high school students, as well as possible.

The tests consisted in a time-taken playing session, with a limit of 30 min in a test scenario. It is expected that the player accomplish the main goal, but it’s considered other possibilities, like the player needs a “little help” or the user leaves the game cause frustration. It’s important to remark that all the sessions were filmed so they can be reviewed after. Users were located in an isolated room, in a computer with the game installed and running. The only other people allowed to be there were the observers as shown in Fig. 4.

The first iteration was made in a black-backed scenario in one single space that can be scrolled horizontally; it was only for testing commands and movements. Further versions included full graphical scenario with doors, elevators, high and low points, objects to be collected and a statue for path indications. Latest version also includes initial and final video sequences and a floating book for command help. All movements are not allowed at the start of game, they are gradually enabled and explained by the help system, so, for example, character can’t move objects before it’s allowed, even if there are objects that can be moved.

Before each test, users were informed about the most important facts of the test: they will play a game, they must self-learn how to move, how to take objects, how to jump and how to reach the goal-point. Also, they were provided with a pre-test questionnaire requesting information about:

• Gender

• Age

• Formal studies

• Game playing experience, including weekly playing time, and some types of games previously played.

During the test the information collected was reported by observers, and includes how the user completed the tasks, and how easily was for him/her. The tasks include some actions like:

• Move (left/right)

• Jump

• Hang up

• Pull

• Push

• Go up

• Other general aspects of the game

After each test, user was prompted for a post-test questionnaire, which requests information about his/her experience playing the game, this includes the facts of:

• How useful was the help and if it was in the appropriate moment

• How appropriate was the command-key layout

• How the character movement was perceived

• How the character reacted to commands

• How easily the tasks (move, jump, hang up, pull, push and others) were made

• Best and worst facts perceived by the user about the character control

• What is the main goal of the game

• What is the user perception about the main character

The first iteration provided useful information about the commands for realize the actions and how quick or easily the user learned about it. This iteration was applied to two young-mid age users.

Second and third iterations used full graphical interface (Fig. 5), and considered one scenario with many rooms, connected by doors. In each room there are objects to be collected, dangerous places, help for commands, moving objects (acting as elevators), and decorative objects. In this scenario, user must distinguish what is useful and what is not, and find the way through the rooms. These two iterations provided very useful information about difficulties previously not seen, and also help and navigation issues. The second iteration was applied to three mid-age users and the third one was applied to a teenager (Fig. 6).

Fig. 5.

First iteration: user test

Fig. 6.

Second and third iterations: user test

The last iteration was made to a group of 24 teenagers. As before tests, this was conducted in computer laboratories with the testing version of the game previously installed. Time was extended to 40 min. Only users and observers were allowed to stay in. The profile of the tested users is showed in the Table 1.

Table 1. Last-iteration user profile by gender, age and declared playing time

4 User Evaluations Results

Each iteration exposed some issues and difficulties found playing the game. There was a place for comments that brought feedback for improvements in the game, and those improvements made changes in later test questions, so improvements in user experience features can be tested.

The first iteration showed confusion about movements. Because they were applied to previously-experienced gamers, the layout was not as they expected. Also, users noticed that the movements are not felt natural, dangers are difficult to see, and also it was difficult to difference between movement and actions like push or pull.

In the second iteration (full scenario and graphical interface), the users also perceived not natural movements but with less intensity than the first iteration. Improvements noticed include more natural hang up and a better help. One user left the game, and the other two ended with a bit of help. In this case, other issues were discovered, most important were:

• Help, even improved, was not easily understood.

• Directions for path are not clear, “help” points to the final destination but not to the nearby path.

• Users don’t know what the objects in the inventory are and how they get there.

• Users have unlimited lives, it was perceived boring for one of them.

• There are checkpoints in some places, but they are not adequately placed.

• It’s very difficult to guess some actions: how to pull a box that is over another one?

• It’s not clear if a long jump requires previous running for taking “impulse”.

• The user wants to make the character run, but it only walks.

• Time for final stage was too short, and don’t left time for the user to think or plan a strategy.

In the third iteration, applied to a teenager, most of the issues were corrected, but the game was not finished successfully, the main reasons include help was still difficult to understand.

The fourth and last iteration was made to a group of teenagers, with all the observations and issues fixed and/or improved. Almost all the issues were fixed, but the “running” feature wasn’t added, instead, there’s a “slow move” and a normal faster move. In this case, they were mostly experienced players (see Table 2). In this iteration users noticed:

Table 2. User perception of the game in the last iteration. In a scale of numbers, 1 means “most difficult” or “never”, 5 means “most easy” or “always”.

• Help was very clear for half of the users, and clear for the rest. No one noticed some issue.

• Path indications were mostly clear.

• Movement commands were easily learned and done, even if they are not equal to other games played before by the users.

• There’s still a perception that the character’s reaction was not natural as it’s expected.

• 11 players ended the game successfully before the expected time limit (30 min).

• Three users understood easily all the facts of the game and finished quickly without external help. Others accomplished tasks in more time.

• Eight users required external help but accomplished the tasks.

• Two users tried to do movements that are not allowed at that time.

• The video sequences helped to understand the story around the main character.

• One player abandoned the game at 26 min.

• Players who completed the game showed an average time of 33,8 min.

• Nobody noticed the need of run.

5 Conclusions

Use of user experience (UX) integrated with software development stages was useful since it provided quick information regarding user experience issues and make the development process fix those issues while improving new features.

Earlier test iterations showed important improvements in user experience, so these features were available quickly in another testing iteration. Because they were applied to previously-experienced gamers, the command layout was not as they expected. Also, users noticed that the movements are not felt natural, dangers are difficult to see, and also it was difficult to difference between movement and actions like push or pull.

In the last version, almost all the issues were fixed. Help and path indications were almost clear all users for the rest, movement commands were easily learned and done, even if they are not equal to other games played before by the users, even if there’s still a perception that the character’s reaction was not natural as it’s expected.

Final iteration provided broader results in target users, and all of them never played the videogame, so their feedback provided very useful information about user experience features and improvements, and also gave the research team a path to add new features.

Video games are not unrelated to the techniques of agile software development and usability techniques on the contrary the application in its development cycle is very beneficial to obtain a better quality of product without neglecting the other aspects of this type of software.