Outcomes assessment pitfalls: challenges to quantifying knowledge gain in a sex education game

Inclusion Criteria and Deployment Structure

MFF_2018 was deployed in Chennai, India as part of research conducted by Dr. Swathi Padankatti and her team from the International Alliance for the prevention of AIDS in collaboration with the U.S. based game development team (Dr. Bertozzi’s group at Quinnipiac University) and Dr. Aparna Sridhar at U.C.L.A’s School of Medicine. Dr. Padankatti and her team identified three schools willing to participate in the study who could provide a total of 419 student players (208 males and 211 female) ranging from 13 to 16 years of age. This was a convenience sample due to the sensitive nature of the game content. Schools were selected based on the research team’s pre-existing relationships with administrators, with whom they had previously worked on AIDS education initiatives and were open to curriculum initiatives relating to sexual knowledge. The fact that these educators and students had previously participated in HIV education initiatives made it likely that the sample audience would have more knowledge about sex and reproduction than the population at large.

For game deployment in each participating school, 30 android tablets and headsets were set up in a school classroom, and groups of students in the target age group successively cycled through to play the game and discuss their experience. Participants were invited to play the game if they desired to do so and could stop at any time. No students were excluded from participation. Groups were not segregated by sex. To ensure comfort and privacy, students were able to move freely around the room to find their preferred space to sit and play.

In the MFF_2017 deployment post-game questionnaires were used for assessment. These were paper forms filled out by students after playing the game, asking students to qualitatively self-assess knowledge gain and provide feedback on the process of gameplay. While these questionnaires provided valuable feedback and indicated high rates of self-assessed knowledge gain, they were not efficient data-collection strategies. Because forms were filled out on paper, response rates were low and it was not possible to link student feedback to specific test-takers. Positive self-assessment of knowledge gain was encouraging, but not a rigorous method for determining game efficacy. The absence of an evaluative framework for the game was the primary motivation for development of the pre-post testing process in MFF_2018.

Once the participants began playing, they were first asked to indicate their sex and age, after which the pre-test was triggered prior to initiating the main game. The post-test, with exactly the same structure and questions as the pre-test, appeared after completing the game.

Given that MFF_2018 was the first field deployment of the pre-post assessment, the deployment team reported issues to the U.S. development team after each play session. The issues were collected and organized into topics. Below we describe the challenges encountered in both the MFF_2017 and MFF_2018 tests and how they were addressed or are planned to be addressed in future revisions of the game.

MFF (original and modified versions of the game are available here: https://osf.io/gtfu5/wiki/home/ (Bertozzi-Villa et al., 2020). The apks can be installed on any Android tablet or phone.). A video of gameplay can be viewed here: https://familyplanninggame.qu.edu/ which includes all the minigames.

Pretest Challenge One: Communication

Discussion of human anatomy and behavior regarding sex and reproduction is problematic in India (Ismail et al., 2015). Many adolescents receive very little information from their parents or teachers due to cultural taboos (Khubchandani et al., 2014). In designing the MFF_2017 game, we were very careful to introduce explicit material slowly and through a process in which it was revealed in context. The game was constructed so that at each point where explicit material is available for the player, the player was asked whether or not they wanted to see it, and then if they agreed, the material was presented in a context that made sense based on the information being gathered.

For example, when players were asked information about when they planned to start dating a possible partner, they were provided with information about the anatomy of the opposite sexⁱⁱ. When they were asked about the age they planned to marry, after consenting (Figure 1), they were given information about how intercourse works via the animation in Figure 2.

Figure 1. Consent screen from Getting Married Milestone.

Figure 2. Still from animation that demonstrates sexual intercourse.

The addition of a pre-test that collects knowledge of human sexual anatomy and reproduction before the game begins required exposing participants to images and terms before we were able to prepare them the way we had with gameplay without assessment. To be as accessible as possible to players at any reading comprehension level, the game includes as little text as possible and communicates most information through graphics, audio and animation. This is especially important when discussing information about sexuality because these terms may not be familiar to students. However, the inclusion of the pre-test introduced a great deal of technical vocabulary in English before the gameplay began. All of the schools included in the MFF_2018 study had instruction in English, but it was unclear if terms like testicles, ovaries, urine and feces were well-understood by players (Figure 3). Although education about sexual functions is technically part of the educational curriculum for all students in India, the content is not actually taught in many schools due to cultural reluctance to discuss sexuality.

Figure 3. Final anatomy screen from pre-test.

Solution: We sought to address the challenge of not startling participants by designing an interface that introduces the sensitive content in the pre-game assessment carefully. Images are largely outlines with enough detail to communicate but not enough to be offensive. Biologic terms relating to specific body parts are shown in limited number on the screen where they belong so even if the player does not know exactly what the term means they can start to associate the correct term with the appropriate body parts. Players are taught the correct terms and their associated body parts when they play the game.

Pretest Challenge Two: Interface Design

A usability issue encountered during the MFF_2017 was that players lacked familiarity with the drag and drop interface commonly used on smartphones and tablets. The design team determined that the 2018 pre-test was the perfect opportunity to teach players how to use drag and drop so that they would be prepared for it when they reached the game. We also determined that we could use the interface design to reinforce the learning of anatomy by clarifying the correlation between the body parts pictured on the screen and their own bodies as explained below.

The key educational content of each milestone of the game is outlined in Table 1.

To assess knowledge of these topics while training students on a drag-and-drop interface, the pre-post test was designed to show male and female figures in outline, with internal organs visible. A series of 14 anatomy questions covering the full scope of in-game content was presented in a sequence of views. Players answered questions in the pre-test by dragging a word representing a concept (usually with an animation to help explain it) to the correct location on an image (Figure 4). The structure and content of pre and post-tests is identical.

Figure 4. Example of pre-test anatomy question.

The assessment was designed to correlate with the way information is delivered in the different milestones in the game. In the MFF_2017 deployment we noted that players had a difficult time understanding where different organs were located in the body and what their functions were. In the pre-post additions, we were careful to depict both male and female bodies as a whole at the start of the pre-test. The view then zooms in to just the abdomens of the male and female bodies. We added the whole person views in the top right and left of the screen so that players could understand which view of the body was presented to them. It is very difficult to understand how organs are laid out relative to other organs. For example, in the female body, it can be difficult to show the positions of the three apertures of the urethra, vagina, and rectum relative to one another. The additional views were added to minimize this confusion.

Our hope was that the layout of the pre-test prior to play would prepare players to approach the anatomy section of the game where they have to drag and drop each body part to its correct location (Figure 5). During the MFF_2017 deployment of the game, it was clear that some players did not understand the difference between the front and side views of the anatomical drawings. In addition to adding the side views in the upper right and left corners of the pre-test, we also incorporated them into the minigame. These views update as each organ is dragged into the correct location in the front view.

Solution: The drag and drop training in the pre-test was successful. Players learned how to use it in the pre-test and used it in the game without the difficulties encountered in the previous deployment.

Figure 5. Screenshot of anatomy minigame.

Pretest Challenge Three : Ensuring Fun and Making Assessment Invisible

During the early stages of the MFF_2017 deployment, teachers stayed in the room during gameplay. They often gave stern instructions on how to behave and ordered students to follow the instructions of the researchers. We realized that this made it impossible for students to experience playing the game as play. Due to the presence of their teachers, students experienced the session like a test that they were required to engage in. To encourage a sense of play, the research protocol was modified early in the first deployment to ask instructors to leave the room during gameplay. Additionally, language was added to the introductory scripts, encouraging students to play the MFF game as a game – they should only do the parts of it that they wanted to, and could stop playing at any time. This protocol was extended into the MFF_2018 deployment where considerable care was taken to encourage a sense of play and remove the pressure associated with the school environment. By introducing a pre-test, however, we recreated the circumstances under which the experience of play was potentially undermined. Students were invited into the room to play a game. However, after they are welcomed to the game, they are presented with an assessment. The 2018 deployment team reported that some students were concerned that they did not have the “right” answer and wanted to be able to go back and correct their previous answers during the pre-test. Given that the Indian system of education heavily relies on test scores and impactfully rewards those who test well, these students appeared very motivated to “do well” as soon as they realized it was an assessment.

Solution: To counter this, the researchers repeatedly stated that they should just answer what they knew and then go on to the game, but this clearly affected the experience. We learned that in future deployments, we need to add more context and less pressure to the pre-test to ensure players understand that they will not be criticized or penalized for not knowing the answers. We can actually leverage their desire to do well by encouraging them to see if they can figure out what they didn’t know in the pre-test during subsequent gameplay.

Post-test Challenge: Adherence

We encountered more serious issues with the post-test. While the transferal of the post-test questionnaire into a digital framework did allow for personalized tracking of results, there were challenges to collecting post-game information. Qualitative feedback from the 2018 deployment team indicated that, when students came to the end of the game, and saw the same screen they had seen earlier for the pre-test, many simply dragged the tiles to “Not Sure” because it was the fastest way to get to the final screen. Others simply put down the tablet which meant that researchers had to exit the player from that game session (with no responses to the post-test questions) to reset the tablet for the next group of students. Due to the fact that we do not know exactly what happened in all the cases where there appear to be random answers to the post-test, we cannot determine how many students actually answered the questions intentionally.

Solution: Because the post-test was not only clearly an assessment, but also not well-integrated into the play experience, many players simply abandoned it. We failed to provide players with a compelling reason to want to engage in the final assessment as part of gameplay, which will be corrected in the next deployment.

Assessment of pre-test scores

The pre- and post-test were identical, each consisting of 14 anatomy questions presented via a drag-and-drop interface with figures. Students received a single point for each question they answered correctly, and no points for incorrect responses. In the results below, test scores are always presented as percentages. A total of 419 students in three schools completed the pre-test and main game. The schools were selected based on scheduling availability and willingness to participate. The researchers from the IAPA had previously worked with these schools on AIDS education initiatives. Across all schools, the pre-test score was 33.5% on average (SE 1.15%), with substantial variation between schools. In particular, students at School 2 (who had sexual education as a formal part of their curriculum) performed significantly better than students at Schools 1 or 3 (two-tailed t-test p<0.001). Pre-test scores were not significantly different between male and female students at any school (Figure 6).

Figure 6. Pre-test scores by school and sex of respondent.

Bars represent the standard error of the mean. N=419.

Across all schools, students scored slightly better on pre-test questions relating to the anatomy of their own sex compared to the opposite sex, but this effect was not statistically significant (Figure 7). For female anatomy questions, 34.4% (SE 1.86%) of female respondents answered correctly, compared to 30.6% (SE 1.6%) of male respondents (two-tailed t-test p=0.13). For male anatomy questions, 33.3% (SE 1.89%) of female respondents answered correctly, compared to 36.3% (SE 1.8%) of male respondents (two-tailed t-test p=0.25).

Figure 7. Pre-test scores by type of question and sex of respondent.

Bars represent the standard error of the mean. N=419.

As shown in Figure 8, the only questions for which a majority of responses were correct were “Where is urine excreted from a male?” (53.0% correct) and “Where does a lining build up to prepare for pregnancy?” (50.6% correct). For eight of the remaining 12 questions, the correct answer received a plurality of responses, but not a majority. The four questions for which the most frequent response was not the correct answer were “Where sperm exit the body?” (plurality answer “Not Sure”, 29.8%), “where menstrual blood is excreted?” (plurality answer “Not Sure”, 27.7%), “The organ that becomes erect before intercourse?” (plurality answer “Vagina”, 33.7%), and “Where urine is excreted from a female? (plurality answer “Vagina”, 35.6%)”.

Figure 8. Pre-test responses by question.

The highlighted bar shows the correct answer in every instance. N=419.

Pre-post test assessment

As described above, assessment of knowledge gain was complicated by the large number of students who did not complete the post-test or who rushed through it, answering “not sure” to all questions (173 students, 41.3% of total). We refer to this group as having a “null” post-test. While it is not possible to assess knowledge gain among those with a null post-test, among the 246 (58.7%) students who did attempt the post-test we find on average a 6.27-percentage-point score gain between pre- and post- tests (95% CI 3.8–8.75, p<0.001, one-sample t test, Figure 9). Scores ranged from 0 to 100 for both tests. In the pre-test, 17 students answered all questions incorrectly while one student answered perfectly. In the post-test, five students answered all questions incorrectly and five students answered perfectly.

A question-by-question breakdown of pre- vs post- test result among those who attempted the post-test shows the largest knowledge gain around topics of intercourse, egg storage, and sperm movement (Figure 10).

Figure 9. Violin plot of score distributions in the pre- and post-test, for those students who completed both tests (N=246).

Points represent means, and bars represent two times the standard error.

Figure 10. Question-by-question comparison of pre- and post- test responses, by proportion, excluding those with “null” post-tests (N=246).

Underlying data

Open Science Framework: Outcomes Assessment Pitfalls: Challenges to Quantifying Knowledge Gain in a Sex Education Game. https://doi.org/10.17605/OSF.IO/WMHCD (Bertozzi-Villa et al., 2020)

This project contains the following underlying data

Extended data

Pre- and post-test data were analyzed and visualized using R version 3.6.0. All code is available from GitHub (https://github.com/bertozzivill/india-family-planning) and archived with Zenodo (http://doi.org/10.5281/zenodo.3822455 (Bertozzi-Villa, 2020))

Data are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).