Student Engagement and Study Abroad

In this study the authors assessed student engagement during a short-term study-abroad program using the National Survey of Student Engagement (NSSE). Data were collected from a group of Canadian undergraduates spending six weeks in Mexico. Their program included a 10-day bus tour, three half-credit courses, and accommodations with local families. The authors administered the NSSE twice: once at the conclusion of the students’ current school year and six weeks later at the end of their study-abroad program. A comparison of responses from the two administrations of the NSSE indicates an advantage, though modest, for the study-abroad setting. Study abroad’s stature in higher education as a high-impact learning activity received moderate support from the data. The authors encourage adminis-trators and researchers to use student engagement and the NSSE to refine study-abroad programs.


Résumé
Dans cette étude, les auteurs ont évalué l'engagement des étudiants au cours d'un programme à court terme d'études à l'étranger, à l'aide de la National Survey of Student Engagement (NSSE). Un groupe d'étudiants canadiens de premier cycle a colligé les données recueillies lors d'un séjour de six semaines au Mexique. Leur programme comprenait un circuit touristique de dix jours en autocar, trois cours d'un demi crédit et l'hébergement chez des familles locales. Les auteurs ont demandé aux participants de répondre au sondage de la NSSE à deux reprises : une première fois à la fin de l'année scolaire en cours, et une deuxième fois six semaines plus tard, à la conclusion de leur programme d'études à l'étranger. Une comparaison des réponses obtenues à chacune des deux séances du sondage du NSSE indique un avantage modeste pour les paramètres de l'étude à l'étranger. En effet, les données obtenues appuient modérément l'importance des études à l'étranger en enseignement supérieur en tant qu'activité d'apprentissage ayant une As the economic, sociocultural, and political forces of globalization become increasingly pervasive, many universities are responding in part by enhancing their study-abroad programs. Proponents of these programs argue that studying abroad has a profound effect on students, and those close to the students say the experience changes them somehow. Researchers, however, have struggled to operationalize these impressions and gather evidence that would demonstrate the value of study abroad. The soundest studies use constructs and instrumentation developed around campus-based educational objectives and settings, but these often miss the point of the study-abroad enterprise and perhaps as a result are inconsistent in their conclusions.
The National Survey of Student Engagement (NSSE) has also been developed around campus-based education; however, it has several qualities that could address current problems in the evaluation of study abroad. Most important, the construct-student engagement-and the instrument-NSSE-foreground the types of educational experiences and outcomes that studyabroad proponents esteem but have been unable to articulate satisfactorily. Moreover, the survey measures a construct developed through decades of influential research in higher education; it is deployed by a growing number of universities in Canada and the United States as the primary measure of the quality of their students' experience on campus; and its psychometric properties have been established over dozens of studies and tens of thousands of respondents.
In this study, we used the NSSE to assess student engagement in a study-abroad program. The study is part of a larger effort to evaluate a long-standing study-abroad program, and it is a provisional exploration of the NSSE's appropriateness for determining the merit of any studyabroad program, communicating the results to discriminating audiences, and guiding the design and delivery of the study-abroad programs.

Study abroad
Study-abroad programs have many forms, even within the circumscribed set offered in formal higher education. Several axes have been identified on which programs vary, including duration (six to eight weeks, one semester, one year), pedagogical approaches (experiential learning, service learning, instructivism), type of accommodations (dormitory, hotel, homestay), entry language requirements, and the extent to which students are compelled to use the local language (Engle & Engle, 2003). The permutations of these aspects make it difficult to provide a concise description of study abroad.
Evaluations of study-abroad programs typically examine changes in students' disciplinary knowledge, professional development, personal development, additional-language skills, and intercultural competence. The last two outcomes have received the bulk of researchers' attention. Across studies, conclusions are equivocal. incidence considérable. Les auteurs encouragent les administrateurs et les chercheurs à utiliser l'engagement des étudiants ainsi que la NSSE pour raffiner les programmes d'études à l'étranger.

The National Survey of Student Engagement
The National Survey of Student Engagement may offer one solution to this gap in research and practice. The construct measured by the NSSE is student engagement, which is a reflection of the extent to which students engage in learning activities that have a demonstrable impact on their intellectual and psychosocial development. Intellectual development includes higher order, reflective, and integrative thinking. Psychosocial development includes personal and social growth, practical competence, and general education. The types of learning activities that engender these two broad categories of development (ctual and psychosocial) can be grouped into five categories: (1) active and collaborative learning, (2) faculty-to-student contact, (3) academic challenge, (4) supportive campus environments, and (5) enriching educational activities. On the NSSE, these are represented in questions such as the following, respectively: (1) How often have you worked with students on projects during class time? (2) How often have you discussed ideas from your readings or classes with a faculty member outside of class? (3) How much of your coursework has emphasized synthesizing and organizing ideas, information, or experiences into new, more complex interpretations and relationships? (4) To what extent are your relationships with other students, faculty members, and administrative personnel friendly, supportive, and helpful? (5) Which of the following do you plan to do before graduating: practicum, internship, field experience, co-op experience, or clinical placement?
Numerous studies have been conducted with the NSSE involving tens of thousands of students from colleges and universities in the United States and Canada. Much of the research to date has been descriptive, with institutions developing baseline engagement data. Building on this descriptive project, other researchers have designed studies that compare student populations, departments and faculties, institutional types, and other categorical factors. Most relevant to our concerns are the studies that explore the pedagogical model that underlies the NSSE. There have been a number of these. Several researchers have documented a relationship between student engagement and GPA, persistence, academic development, knowledge acquisition, analytical and problem-solving skills, and self-esteem (Carini, Kuh, & Klein, 2006;Kuh, Cruce, Shoup, Kinzie, & Gonyea, 2008;Kuh, Kinzie, Buckley, Bridges, & Hayek, 2006;Kuh, Laird, & Umbach, 2004). Other studies have begun to establish relationships between the single dimensions of the NSSE and favourable outcomes. One study found that students engaged in active and collaborative learning activities were more likely to be actively involved in a variety of educationally purposeful activities, and more likely to report greater gains associated with these experiences (Kuh et al., 2004). Another found that informal student-faculty interaction was positively correlated with student learning and development (Kuh & Hu, 2001).
The specific value of study-abroad programs has been examined in one NSSE study that found that students' participation in four high-impact activities is strongly associated with gains in deep learning, general education, personal and social development, and practical competence (Gonyea, Kinzie, Kuh, & Laird, 2008). The activities were study abroad, research with a faculty member, capstone projects, and service learning. Explaining the relationship between these activities and study abroad, the authors suggest that study abroad increases the odds that students invest time and effort in their studies, interact with faculty and peers about substantive matters, get more frequent feedback, and discover the relevance of their learning through real world application.
The purpose of this study was to evaluate a study-abroad program in terms of student engagement. A successful program, we determined, would be one in which students' ratings of deep learning and general education would be higher for their study-abroad experience than for their on-campus experience.

Research Design
As an initial exploration of this question we developed a descriptive study in which a group of students provided information on engagement in their on-campus and study-abroad experiences.

Participants
We collected data from each of the 25 students participating in a study-abroad program offered by the University of Calgary. The program included a ten-day bus tour of the host country, accommodations with local families, and three half-credit courses (introductory courses on Spanish language, Mexican culture, and archaeology). The participants were undergraduate students from a variety of faculties. Eighty-eight percent of the students were female, and an equal percentage of the group were ages 20 to 24.

Data collection
The participants completed a version of the NSSE that we modified for our purposes. Our specific interest was in the sections of the instrument that assess deep learning, active and collaborative learning, faculty-to-student contact, level of academic challenge, enriching educational activities, and supportive environment. (All items in our survey are reproduced in Tables 1 through 4.) We removed any NSSE items that did not assess these processes. In total, each form of our instrument contained 27 items. We administered one form of the NSSE on campus prior to the students' departure and the second form in the field during the final days abroad. We prefaced each administration by directing the participants to consider their experiences on campus or during the study-abroad program, respectively.

Data analysis
We followed the conventional procedure for analyzing and reporting NSSE data. This involves organizing the items into their subscales and reporting the percentage of students who select each of the four response categories (very often, often, sometimes, never) for each item. For clarity of presentation, we collapsed the four categories into two (frequently, infrequently).
Because this is an evaluation of an existing educational program, we did not strategically sample students; instead, we collected data from each student who participated in the program. Therefore, we do not offer inferential analyses. We report effect sizes for each subscale using relative risk as an index of effect measure for this set of ordinal and nominal data.
Permission to conduct this study was received from the University of Calgary's Conjoint Faculties Research Ethics Board.

Results
In this section we contrast the students' responses to the two administrations of the NSSE, the first in which they focused on their on-campus experience, and the second in which they focused on their study-abroad experience.
Tables 1 through 4 present the percentage of students across response options on each item of the four NSSE subscales that provided the focus of this study. The tables also present the effect size. Higher order learning is represented on the NSSE with four items that prompt students to consider the extent to which learning activities required them to analyze ideas, synthesize information, make judgments, and apply theory. Fewer than half of the students reported that such requirements occurred frequently; when they did arise, they were somewhat more likely to be invoked by their on-campus learning activities (see Table 1).
Reflective learning is represented on the NSSE with six items that prompt students to consider the extent to which learning activities required them to examine their beliefs, understand others' perspectives, change their understanding, discuss complex issues, personalize their learning, or engage in serious thought. A majority of students reported frequent engagement in reflective learning activities in both settings; however, reflection was more frequent during study abroad (see Table 2). Integrative learning is represented on the NSSE with five items that prompt students to consider how often their learning activities required them to integrate information from various sources, include diverse perspectives in discussions or assignments, synthesize ideas from different courses, and continue scholarly discussions with instructors and other students outside of class. The students reported that such requirements were more likely to be invoked by their on-campus learning activities (see Table 3). Including diverse perspectives and integrating ideas were more likely to occur in the study-abroad context; the other activities were slightly more likely in the on-campus context.
General education is represented in the NSSE with 12 items that prompt students to consider the extent to which learning activities prepare them for a broad set of tasks they will  Table 4). For nine of the 12 items, a majority of students agreed that their experience, either on campus or abroad, contributed to their general education. Generally, students were more likely to attribute their acquisition of these 12 items to the study-abroad experience, with notable differences in developing a personal code of ethics, understanding people of other racial and ethnic backgrounds, and acquiring job-or work-related knowledge and skills. Neither setting instilled a civic responsibility to contribute to the welfare of their community or to vote. 1.03 * Calculated as the relative risk of responding often and very often to the integrative learning item when thinking of the study-abroad experience versus the on-campus experience. Relative risk estimates less than 1 mean that students were less likely to engage in integrative learning during study abroad than on campus. Contributing to the welfare of your community 39 29 0.73 * Calculated as the relative risk of responding agree and agree strongly to the general education item when thinking of the study-abroad experience versus the on-campus experience. Relative risk estimates greater than 1 mean that students were more likely to engage in general education during study abroad than on campus.

Discussion
The purpose of this study was to assess the merit of a short-term study-abroad program. Our criterion for determining merit was student engagement as it is operationalized with the NSSE.
There was a pattern of results favouring the study-abroad experience: reflective and integrative learning and general education were associated more frequently with study abroad, higher order learning with on-campus experience. However, most of the differences were modest. Participants reported levels of engagement during their study-abroad experience that were similar to levels in class, on campus.
Our results are consistent with those of several others who find limited empirical support for short-term study abroad in higher education (Asay et al., 2006;Bataller, 2010;Black & Duhon, 2006;Davidson, 2010;Dekeyser, 2010;Freed et al., 2003;Douglass & Jones-Rikkers, 2001;Emert & Pearson, 2007;Lindseth, 2010;Martinsen et al., 2010;Tajes & Ortiz, 2010;Wilkinson, 2002).An examination of these reports suggests that the lack of an unmistakable difference may be a gulf between the potential of study abroad, which captivates proponents, and the actual effect that is observed and reported by researchers. The potential is students engaging in goal-directed behaviour-linguistic, cultural, disciplinary, personal, or professional goals-amid the complexity of their subject matter unfolding in real time. What actually happens, in those instances when the benefit of study abroad is equivocal, is students circumventing immersive, goal-directed activity. The students in our program, for instance, were not immersed in Spanish language, Mexican culture, or Nahuatl architecture. Though their coursework required them to, for instance, conjugate Spanish verbs and view slides of heritage buildings, they spent the bulk of their time travelling in a tight group, moving from the classrooms where they passed much of their days to the Internet cafés at night to work on assignments. Avoiding any real need to grapple with intercultural issues, they were in continual contact with their friends and family back home via Facebook, email, and text messaging.
There are several reports of this type in the literature (Knight & Schmidt-Rinehart, 2002Schmidt-Rinehart & Knight, 2004). Researchers report little interaction between studyabroad students and locals, even among students living with host families. Perhaps the most poignant example comes from a student who reported, "I was just so surprised that you could be in France for a month and really not speak French that often. I mean, I probably spoke about maybe three sentences a day in French with my family" (Wilkinson, 1998, p. 39 descriptions and our observations seem more characteristic of tourists on vacation than of students engaged in an educational-purpose experience.
Given the widespread difficulty in converting the potential of study abroad into educational impact, some suggestions for practice are warranted. Among the few studies in which an educational impact is reported, some commonalities exist. First, measurable benefits occurred almost exclusively in programs with a duration of one semester or longer. This excludes short-term study abroad, which is commonly understood to last no longer than six weeks. Unfortunately, for many universities and their students, extended sojourns are not viable. Therefore, other issues come to the fore.
Students who benefit from study-abroad programs are thoroughly immersed in the experience. They speak the host language, not their first language; and they spend the bulk of their time with members of the host country, not with others in their study-abroad group (Brecht & Robinson, 1993;Hernandez, 2010;Isabelli, 2001;Knight & Schmidt-Rinehart, 2002Martinsen et al., 2010;Schmidt-Rinehart & Knight, 2004;Twombly, 1995;Wilkinson, 1998). Unfortunately, few authors have identified methods to ensure student immersion. The program we studied, however, seemed designed to discourage immersion. To receive accreditation from the university-and legitimacy in the eyes of departmental faculty-the program's developers were forced to overload the brief study-abroad tour with conventional, classroom-based coursework involving didactic instruction and objective, standardized forms of student assessment. These requirements interfered with the students' and instructors' ability to seize many rich learning opportunities present in the setting and circumstances.
The NSSE could also be an important tool in subsequent research. The construct it measures, engagement, connotes something similar to immersion; its psychometric properties have been investigated and documented; its underlying model has emerged from decades of influential research on higher education; and it operationalizes many of the purported benefits of study abroad. Others may find, as we did, that the NSSE data point to specific strengths and weaknesses of particular activities, and they do so in a manner that facilitates revision.