Who Selects an Online Class Over the Same Course Face-to-Face? And Who Learns More? Results from a Mixed-Methods, Quasi-Experimental Study of Teaching the Sociology of Work and Family

Recommended Citation Parcel, Toby L.; Radu, Monica Bixby; and Gonzales, Laura Fitzwater (2018) "Who Selects an Online Class Over the Same Course Face-to-Face? And Who Learns More? Results from a Mixed-Methods, QuasiExperimental Study of Teaching the Sociology of Work and Family," The Journal of Public and Professional Sociology: Vol. 10 : Iss. 1 , Article 6. Available at: https://digitalcommons.kennesaw.edu/jpps/vol10/iss1/6


Introduction
Online teaching in the United States has grown substantially over the last decade, and the 21 st century will likely see even further increases (Allen and Seaman 2010). Scholars estimate that in recent years, the percentage of students taking at least one online course during their college careers has nearly tripled (Allen and Seaman 2013), and the number of students not taking any distance education courses has steadily declined (Allen and Seaman 2016). Deming et al. (2015) find that in 2013, twenty-five percent of undergraduate students took at least one online course, while more recently Bettinger et al. (2017) estimate that during students' college careers, approximately one-third will take an online course.
Because of growing demand for online courses, colleges and universities have responded by offering a variety of options, including programs that combine both online and traditional learning instruction. For example, Deming et al. (2015) find that among non-selective public colleges and universities, in 2013 nearly 20 percent of students combined both online and face-to-face courses, taking at least one online course. There is also an increase demand for programs operating exclusively online and some large public universities now offer fully online programs (Deming et al. 2015). They also find that in 2013, for-profit universities, such as DeVry, Kaplan, and the University of Phoenix enrolled more than half of their students in fully online programs.
In this paper, we analyze the student learning and background characteristics of multiple sections of an upper-division undergraduate work and family sociology course, two taught face-to-face and four offered completely online. Our paper is innovative because the learning environments across the two delivery modes were strictly controlled, with the same instructor in all six sections, the same assignments, and the same grading standards. We also collect both qualitative and quantitative data. After reviewing relevant literatures, we provide detail regarding the academic and social background characteristics of the students by delivery mode, as well as qualitative information reporting their motivations for selecting the section they chose. This is important because if students strongly "self-select" into respective sections, these characteristics must be controlled as we assess whether learning outcomes differed. With this information as context, we then address the question of whether students' learning outcomes differed by delivery mode, as well as what student and course characteristics result in better performance in the course.

The University Context
Our university is classified as R1, highest research activity, by the Carnegie Classification System. 1 It is a land grant institution located in an urban area of the southeast and is one of two flagship universities in the state. Total enrollment exceeds 34,000, with about two-thirds being undergraduate enrollment. The institution offers 106 bachelors', 104 masters' and 61 doctoral degrees. Its endowment is over 1.1 billion dollars and it employs over 2,300 academic staff. By state law, eighty percent of undergraduate students must come from inside the state; about 50% of undergraduate applicants are admitted. As of 2016, the undergraduate population was 45% female and 28% nonwhite. Forty-four percent of undergraduates receive need-based scholarships or grants, and the six-year graduation rate was 76%. Also of note, in recent years our state has experienced substantial population growth. To keep up, the university has projected that some future student demand will be handled via online instruction. This model assumes students would enroll in some face-to-face courses and others delivered online (Gabriel 2010), which means that in any given semester, many of our online students are likely drawn from the traditional student population.

Are There Learning Differences by Delivery Mode?
A key question is whether students enrolled in classes that differ by delivery mode have differential learning outcomes. Although academics have worried that online students will not learn as much as students in traditional classes, some recent literature, not necessarily in sociology, has been reassuring. In a meta-analysis of experimental and quasi-experimental studies comparing levels of learning across delivery modes for K-12 and post-secondary students, Means et al. (2009) reported that online students or those in blended settings had better learning outcomes than students who took the same classes face-to-face. Sitzman et al.'s (2006) metaanalysis of 96 studies suggested similar conclusions. Zhang et al. (2004) found that remote learners can outperform traditional students, possibly owing to the online students being able to repeatedly view entire lectures online and participate in online discussions.
Other studies found no differences by delivery mode. For example, McFarland and Hamilton (2005-06) found no differences in course grades or student satisfaction between traditional and online sections of a business class. Likewise, in a study of introduction to sociology courses that compared a traditional section with a video-conferencing section, there were no differences in attendance or exam scores between the two groups (Koeber and Wright 2008). In a study of in-class and online discussions for pre-service teachers' recall of concepts, both types of discussion produced similar results (Ng and Cheung 2007), while York (2008) found no differences in graduate-level social work students' course grades, gain in self-efficacy, or in course satisfaction by delivery mode. Parkhurst et al. (2008) found no difference in multiple-choice test scores among engineering students by delivery mode, although the face-to-face students fared better on essay exams. Their qualitative evidence suggested that students in the face-to-face sections appreciated the chance to interact with instructors and other students; they also had access to visiting lecturers whose presentations were not available to online students. These findings suggest a combination of differential resources and self-selection may have been factors favoring face-to-face students' development of higher-level thinking skills, including analysis and synthesis.
In view of this mixed evidence, additional questions remain. In upper division sociology courses where there is more emphasis on analysis and critical thinking, do levels of learning differ by delivery mode? It could be that the face-toface setting is more important in helping students develop these higher-order learning skills. For example, Parkhurst et al. (2008) argue that instructors may be more important to learning in such courses. It may also be true that levels of learning may differ by delivery mode for some assignments, but not for others.
These studies also vary considerably in how controlled they were, thus hindering comparability. It is understandable that for meta-analyses, the authors would be looking for similarities and differences across courses from a variety of disciplines where there is understandably weak comparability in assignments, instructional goals, and instructor practices. However, some smaller studies fail to provide enough detail regarding courses taught and instructor practices (Kaupp 2012;Xu andJaggers 2013, 2014). Other studies are strictly controlled by delivery mode, but do not speak to sociology specifically (McFarland and Hamilton 2005-6;York 2008;Zhang et al. 2004).
In contrast, Driscoll et al. (2012) reported no learning differences across course delivery mode in an introduction to sociology course with strong controls for instructor, course material, and assessment. Koeber and Wright's (2008) study had similar advantages in analyzing introductory sociology courses. However, these studies are also limited because they only examined introduction to sociology courses. Overall, there are relatively few strictly-controlled studies that allow us to make comparisons of learning levels across delivery modes at the post-secondary level (Means et al. 2009) and in sociology in particular. We add to the body of knowledge by analyzing an upper-division sociology course while also controlling for the learning environment across sections.

Are Students Who Take Courses Online vs. Face-To-Face Entirely Different Groups?
A related issue is whether students self-select into different classroom settings. This is an important question because if disproportionately stronger students select face-to-face classes, then student strength, not delivery mode, might account for performance differences favoring face-to-face classes. Without the ability to randomly assign students to delivery mode, we believe it is very important to understand descriptive differences in who takes which type of class, and investigate whether these factors affect levels of learning.
Some have held the stereotype that online students are "non-traditional," specifically, that these students are older, likely to be working full-time, and may live some distance from the institution in which they are enrolled. "Traditional" students, in contrast, are assumed to live on campus or close to it, attend school full-time, and be in the 18 to 22-year age range. Although they may work parttime, some assume that progress through college is their main priority and that faceto-face instruction is what they both expect and desire. If the two pools of students are different, then this is important information for instructors as they prepare to meet the possibly unique needs of different groups of students.
Studies have varied in the extent to which they analyzed the characteristics of students across sections using different delivery modes. For example, in Koeber and Wright's (2008) quasi-experiment, the authors found no differences in student characteristics such as student age, percentage female, race, and grade point average (GPA) by section. McFarland and Hamilton (2005-06) did not report differences between their two groups of students, who had self-selected into the sections they compared. York (2008) compared student characteristics including interest in administration, years of social work experience, and Miller's Analogies (MAT) scores, and found few differences. Parkhurst et al.'s (2008) study of a sophomore-level class reported no data on student characteristics. Thus, only a few studies compare background characteristics of students and several report no differences in student characteristics by delivery mode. Even then, the sections varied in terms of instructors (Parkhurst et al. 2008) and other aspects of the learning environments, over and above delivery mode. Instructors cannot typically randomly assign students to sections, thus making analysis of possible student background differences owing to self-selection very important. Our study innovates by describing students in terms of a variety of characteristics, including personality, time allocation, and student perceptions of their learning styles. These data enable us to show in detail how students across sections are similar and different.

Course Design
During the fall of 2009, the first co-author taught two sections of a 400-level sociology course in work and family, one face-to-face, and the other online. Students had the option to enroll in either course beginning in the spring of 2009 and continuing through the first few weeks of the fall semester. The sections covered exactly the same material, readings, and assignments. The instructor met face-to-face with the traditional class twice per week to cover the material and guide discussion, while the online class engaged in online discussion of these points on a weekly basis. We note that in 2009-2011, the costs for the two classes differed; while tuition was the same, students who enrolled in the online class paid an additional fee of $441 per online course. Thus, there was a financial disincentive to take the online as compared with the face-to-face class.
The instructor conducted each section in the same manner, allowing for delivery format differences. Reading assignments for each class consisted of five books covering the history of families in the United States, case studies of work and family dynamics in 20 th century organizations and occupations, work-family conflict, and analysis of the working poor in the United States during and after welfare reform in the mid-1990s.
For each section, both the midterm and final were open-book and open-note essay exams. Students were required to complete four short written assignments and a twelve-entry journal, where each entry required up to two pages of writing that connected course concepts to real-world events. Both sections were also graded on class participation, which consisted of an asynchronous online discussion for the online section (see An et al. 2009;Jaffee 1997) and an in-class discussion for the face-to-face section. Each section had the same quantity of opportunities to contribute to discussion; all assignments were graded by the same rubrics.
The instructor continued to offer this course in following years. She offered the online class during Fall, 2010, Fall 2011, Fall 2012, and Fall 2013. She offered the course face-to-face during Spring 2012. All course requirements and grading practices were identical to those described for 2009. We could not randomly assign students to respective sections, which would have rendered our study a true experiment; however, comparability of instructor, assignments and grading standards means our study is a quasi-experiment.

Study Method
Students voluntarily participated in the study and we followed all protocols required by our Institutional Review Board. We offered all student participators 10 extra credit points for filling out our questionnaires; students who declined participation were offered an alternative 10-point extra credit activity.
We first obtained student background information from our department and devised a survey tapping other variables such as age and allocation of time to work, school, and other activities (full survey available upon request). Our personality measure, the Big Five Inventory (BFI) (John and Srivastava 1999), was designed to tap differences along an introversion-extraversion scale. We reasoned that students who chose the face-to-face sections of the class might be more extraverted or in need of in-person interaction, while online students might be less interested in such interaction or even fearful of it (Ng and Cheung 2007).
We conducted a pre-and post-test of substantive learning, composed of ten multiple-choice questions about work and family life. To measure levels of learning in other ways, we compared students' overall course grades, assignment grades, and the differences between the pre-and post-test scores, which is a measure of factual learning. Recognizing that using course grades to reflect levels of learning may be controversial, we note that several studies reviewed above also use grades to reflect levels of learning (McFarland and Hamilton 2005-06;York 2008). Several also used pre-and post-test scores (Parkhurst et al. 2008;York 2008) as we do. Student retention was in the 95-100% range in all the classes (see Means et al. 2009 for concerns about online dropout rates). In addition, our focus on detailed analyses of descriptive differences by delivery mode reduces concern that differential drop-out would lead to differential learning. Table 1 shows the measurement of all independent variables included in analyses. Table 2 provides descriptive data comparing the characteristics of students by delivery mode. The data suggest that online students were, on average, older, more likely to be living with a partner, have children, work more hours, and take more credits online. Face-to-face students were more likely to be taking more credits face-to-face. However, online students took an average of close to 8 credits faceto-face, suggesting considerable overlap in course-taking patterns. Thus, "online" students are in many cases also taking courses face-to face, suggesting they may live on or close to campus and take a few online credits to supplement an otherwise face-to-face schedule. Students were similar in personality, with the exception that there is a trend level finding that online students are more extroverted, contrary to our initial expectations. The students were also similar in total activity hours, GPA, hours attempted, grade level, and distribution of majors. Overall, the stereotype that traditional and online students are being drawn from substantially different populations is not supported.  Despite these findings of a few background characteristic difference by delivery mode, Table 3 shows that learning outcomes were mostly similar. The one exception is that online students earned higher participation scores than did faceto-face students. This, in combination with the trend-level finding that the extraversion score differences favoring online students, suggests lack of support for the idea that students select into face-to-face classes in order to engage in class participation. In two open-ended questions, we asked students to describe why they chose the online or face-to-face section of the course. The questions were worded as follows:

Student Background Characteristics
Next, we would like to know why you chose this section. This class is offered in two different sections, a face-to-face class and an online class. You chose to take the online section. Why did you choose the online section? Our second question was: Why did you NOT choose the face-to-face section?
Most students gave one or two reasons for their section choice. Following Charmaz (2006), we use line-by-line coding for each student's response. This method allowed us to develop a coding scheme and identify patterns based on students' points-of-views, and it allowed us to better understand why students chose to take the course face-to-face or online in their own words.
We analyzed students' responses to help identify prominent patterns regarding students' motivations for their course delivery mode preference. Overall, we found two distinct patterns. Specifically, students who chose the face-to-face section indicated they were seeking interactions in the physical classrooms with both their classmates and the professor. Students who chose the online section reported doing so because it allowed them flexibility, which was important due to work, school, and family obligations.
Among students who chose the face-to-face section, we found that students discussed how they were seeking social interaction through (1) class discussions, (2) interactions with peers, (3) engaging with the material, and (4) having quick access to the professor. Three students wrote: I enjoy hearing input from other students and being able to express my opinions and gain feedback. I think the classroom is important for really engaging the material.
I believe that by going to a classroom setting it allows you to become more involved in the topic, and are more easily able to ask questions to the teacher and other students.
I feel that I would get more one-on-one time with the teacher and feel more engaged in the class and its material. I also like the structure of going to class (physically) and learning.
In addition, students described their own learning styles. They indicated participating in a face-to-face class made it easier to understand the material and made learning more enjoyable. They believed they learned more. They also discussed how they thought they would not be motivated to participate in an online course because of their lack of self-discipline or forgetfulness. Three students stated: I feel that I learn better with a professor teaching me face-to-face than from a computer screen. Also it is easier to fall behind in online classes because you forget to check the website, whereas if you come to class you are reminded of the upcoming events and assignments.

I chose the online class because over the last few semesters I have been working full-time and have only been taking DE classes. This option is easier for me to do school readings and assignments around my work schedule versus going to campus. I work full-time and having online classes helps me fulfill my work schedule.
The second most frequently mentioned reason was flexibility with their school schedule. Despite enjoying face-to-face classes, several students were limited by their class schedules and needed to take the online section in order to fulfill university requirements. Two students said: [I took the online class] because my classes overlapped for a 15 minute period with the face-to-face class. I prefer not to take online classes, but needed the credit.

Many of the remaining sociology classes that I need overlap in times. It was a matter of being limited by classes being offered and the time I have available.
Another difference between the face-to-face and online sections was selfreported learning styles. Whereas several students in the face-to-face section indicated that they needed the classroom environment to learn better, several students in the online section believed they would learn more efficiently and effectively through distance education. Two students stated:

I chose the online version because I do better in online classes where I can do the work on my own schedule and around my other activities like work. I concentrate on the material better when I'm by myself and not worried about the other people in the class. I chose to take an online class because it best fits my schedule. It allows me to work at my own pace. I try to take one online class per semester.
Additionally, several students in the online sections mentioned family obligations as a reason for selecting the online section. No students in the face-toface sections suggested their families were reasons for choosing to take the course face-to-face. For example, three online students said:

I moved home this semester to help take care of my four-year old sister and to help pay some bills while my mom had surgery.
Taking the online class allows me to spend more time with my two-year old daughter.

Because I work full time, have a family and the online class fits my schedule (work and family) better.
Overall, the reasons students gave for choosing their section were different. Students in the face-to-face sections reported that classroom interaction was vital for learning the material and for motivation. Students in the online sections indicated that flexibility was the most important reason for taking the course online, whether for work, fitting with other classes, or because of family obligations. Instead of arranging other life matters around school, students who chose to take the course online desired to arrange school matters around life. Knowing due dates in advance allowed them the flexibility to complete much of the work at their own pace. No students in the face-to-face sections mentioned that flexibility was important.

Quantitative Plan of Analysis
Our multivariate models investigate whether any learning differences in our work and family sociology course by delivery mode are a function of background characteristics, personality, and academic variables. Because our descriptive results suggest few learning differences, we confine our results to two dependent variables: overall course grades (Table 4) and class participation score (Table 5), the latter result showing stronger performance for online students. Each of these tables shows the zero order effects of delivery mode in Model 1, followed by the additional of background and personality characteristics in Model 2, followed by the addition of academic variables in Model 3.  .476 ***p≤.001, two-tailed, **p ≤ .01, two-tailed, *p ≤ 0.05, two-tailed, †p≤ 0.10, two-tailed.    .416 ***p≤.001, two-tailed, **p ≤ .01, two-tailed, *p ≤ 0.05, two-tailed, †p≤ 0.10, two-tailed. We also tested for statistical interaction. We added five interaction effects to the full additive model (Model 3) predicting overall course grade; these results are shown in Model 4. These include interactive effects between (1) age and faceto-face section, (2) living with a partner and face-to-face section, (3) total activity hours and face-to-face section, (4) total employment hours and face-to-face section, and (5) GPA and face-to-face section predicting overall course grade. We evaluated these same interactions in predicting class participation.
We tested these statistical interactions for several reasons. Prior literature primarily examines the additive effects of both student characteristics and course delivery mode predicting various student learning outcomes. Yet, we suspect there is a more complex relationship between student characteristics and course delivery mode predicting our two course learning outcomes. We expect that student characteristics may affect student learning outcomes differently depending on course delivery mode. These complex interactive relationships are relatively unstudied, which suggests a need for testing the interactive effects noted above. Table 4 allows us to understand more about what background factors predicted success in overall course grades. We already know that delivery mode does not matter, but our analyses shed light on what other characteristics are important. We expected that students who had children, who participated in more activities and/or who had longer work hours would find it more difficult to spend time on coursework, thus resulting in lower levels of course performance. As expected, Model 2 suggests that whether or not students have children and the length of both activity hours and employment hours all negatively affect academic performance. The negative effect of having children weakens in Model 3, as does the employment hours effect. This is due to the strong positive effect of GPA, and to some extent, the positive effect of hours attempted. This latter finding may appear counter intuitive, but it appears that students who attempt more hours have the motivation and drive to complete those credits successfully (see Szafran 2001 for similar findings).

Predicting Overall Course Grades
Model 4 investigates tests for interaction as described above. We found a positive trend-level interaction effect between total activity hours and face-to-face section (p < .10 level, p < .052) predicting overall course grade. These findings suggest that even though the additive effect of activity hours on grades is negative, there may be a tendency for students who spend more time engaged in other activities and take the course face-to-face receive a slight boost to their grades. It may be that the motivation of these students is higher, thus resulting in better grades as well as in more extracurricular/community activity. None of the other interaction effects were statistically significant.

Figure 1. Class Participation Score by Total Employment Hours Per Week and Course Section
Table 5 uses a similar analytic strategy to predict class participation scores. Recall that face-to-face students participate less than online students, which is shown in Model 1. This finding is maintained in Model 2, where we see a trend-level finding for activity hours to depress participation and a statistically significant negative effect of employment hours on participation. In addition, there is a trend-level finding for those high on the personality trait of agreeableness also participate less. Model 3 suggests that GPA is again a strong predictor of class participation. The negative effects of activity hours and agreeableness are strengthened but the effect of employment hours disappears. Interestingly, taking more online credits had a tendency to depress class participation. In Table 5, Model 4, we report the same interactive model as Table 4, Model 4. We found that the interaction effect between total employment hours and face-to-face section is statistically significant (p < .05), predicting class participation score. Figure 1 shows this interactive relationship graphically. In the face-to-face sections, total employment hours had no bearing on participation scores, while for students taking the course online, class participation declined as work hours increased, which likely reflects a tradeoff between engaging in online discussion and working more hours. None of the other interaction effects were statistically significant (p < .05). Our university dropped the cost differential for online course delivery in 2012. We re-ran all our analyses to evaluate whether the year in which students took the course made any difference in our findings. We found that year the course was taken had no effect on our learning outcomes. We discuss this finding further below.
We recognize the strong effect that GPA had in predicting both course grades and course evaluations. To evaluate whether this variable had a notable empirical effect on the remaining predictors, Table 6 shows Models 3 and 4 from Table 4 (predicting overall course grade) and Table 5 (predicting class participation score). In Model 1, we again see the negative effects of activity and employment hours on course grades, as well as the positive effect of conscientiousness; Model 2 shows that none of the interaction effects were significant. Model 3 show the strong positive effect of taking the class face-to-face, and the negative effect of activity hours on class participation. Model 4 shows no interactive effects were statistically significant. Despite some shifts in trend-level findings across the specifications with or without GPA, we conclude that including GPA in Tables 4 and 5 did not substantially change our findings.  .199 ***p≤.001, two-tailed, **p ≤ .01, two-tailed, *p ≤ 0.05, two-tailed, †p≤ 0.10, two-tailed. Table includes unstandardized coefficients, standardized coefficients in italics below and standard errors in parentheses.

Discussion
We have conducted a quasi-experimental study in which we compared learning outcomes between online and face-to-face upper-division sections of a sociology of work and family course. Because the instructor and course requirements were the same for all sections, our study enables us to make better inferences regarding whether course delivery mode is consequential to learning outcomes compared to studies where course requirements and instructors differed by section. In addition, our study is relatively unique within sociology in that the course we studied was at the junior-senior level. This contrasts with a larger number of studies focused on delivery mode differences at the introductory level.
Our findings suggest no learning outcome differences by delivery mode, which is useful information for faculty and department heads who may have the opportunity to offer online classes, but worry that such strategies place student learning at risk. Our findings suggest that with appropriate care, such worries are misplaced. These findings also reinforce the reality that students can build higher order analytic skills as reflected in essay exams and longer written assignments in online learning environments. They demonstrate that when course learning objectives include not only mastery of factual material such as found in introductory sociology, but also strong writing and analytic thinking, the delivery mode did not make a substantial difference in student performance. Given that such course objectives are present in other higher level liberal arts classes, these findings suggest that delivery mode may not be a critical factor in determining levels of learning for undergraduate students in similar classes, in sociology as well as related disciplines.
A common perception in academia is that online students are older and nontraditional, more likely to be employed, possibly full-time, and likely taking a modest course load to pursue a degree over many years. Our findings dispute this stereotype. We found that although there are some differences in terms of online students being older, more likely to be partnered, have children, and work more hours, many other characteristics are the same. These similarities include GPA, most personality characteristics, extracurricular/community activity hours, grade levels, and major distributions. Our face-to-face students were taking an average of almost eight credits online in the semester they participated in this study; this indicates that many students elect to take both traditional and online courses simultaneously. Thus, online students are not drawn from entirely different populations when compared to our face-to-face students. This is important information for all instructors as they prepare to teach upper division students in courses such as the one we studied. We pursue this point in more detail below.
Our study is also innovative in that we combined qualitative data with our quantitative data on learning outcomes. Specifically, our qualitative data reveal why students selected the delivery mode they did. Results suggest that online students value the temporal flexibility that online classes afford. Online students need school to fit into their work and family lives more easily than face-to-face students, who may have a greater capability to fit work and life into school. Students perceived learning styles as well as work and family obligations also played a significant role.
Our findings have implications for instructors who teach upper division online courses. First, such instructors should not assume that enrolled students are necessarily non-traditional, i.e., older, employed full-time and taking a small course load each semester. Although we did have some of those students, they did not predominate. Instructors should use brief introductory exercises to have students explain who they are, whether they are working and for how many hours, and some indication of outside activities. This will sensitize instructors to issues of time management that may be relevant for student success. Overall, such information would be useful to instructors so that classes can be appropriately aimed at what is likely to be a mix of students, both traditional and non-traditional.
Second, our qualitative findings have implications for instructors who teach both online and face-to-face. In college or university settings where students have a choice of delivery mode, students taking face-to-face classes appear likely to be eager for instructor feedback and reliant on in-person reminders about upcoming assignments and evaluations. In contrast, those taking courses online must be self-motivated enough to progress through course materials without these reminders. These findings suggest that to the extent that students are aware of their individual learning styles and have a real choice of course delivery mode, self-selection may be operating to at least somewhat differentiate students across such characteristics.
Our findings did suggest that face-to-face students participate less in class than do the students in the online delivery mode, while those students with higher GPAs participate more. However, we also find that as weekly employment hours increase, online students experience a decrease in their class participation scores. This suggests that although flexibility may be a key reason students choose an online section, online students' participation scores may suffer if they engage in long work hours. This reinforces our suggestion that instructors be aware of students' work and activity obligations so that they can appropriately counsel students whose class participation is suffering. However, this recommendation does not imply that instructors should be modifying course requirements or standards in ways that interfere with the goal of treating all enrolled students the same in a given course.

Limitations
We believe that the institutional context may have been a partial factor in producing these findings. For example, we may be attracting students who are more capable than those institutions that are less selective, but also, we may attract students who are less capable than those enrolled at institutions that are more selective. It may be that the more selective the institution, the less likely there are to be differences in student learning by delivery mode. This is a hypothesis worthy of further research. In addition, we are located in an urban area in the United States. Many of our students both need and have the opportunity to work during the semester. If we had conducted the same study at a rural college, in another country, or at a university with a lower proportion of affluent students, there might have been more striking differences in the backgrounds of students by section, and possibly in learning outcomes, as well. This makes it important for other instructors in different college and university settings to conduct controlled studies such as this one in order to evaluate the external validity of our findings.
We recognize that we produced our data over several years, i.e. between 2009 and 2013. As noted above, we did evaluate the effect of year of enrollment on our findings, and found that this was not statistically significant. Courses at the 400 level, typically taken by juniors and seniors, are not always offered each semester. In our case, the online version our course was only offered during Fall semester, and the face-to-face version was offered only occasionally. Thus, it took several years for us to produce enough data that we could conduct our analyses. This reality may explain why there are far more studies comparing online and face-to-face learning outcomes for lower division courses, which are offered more frequently. This means our study adds a different element to the literature specifically because it does allow us to compare levels of learning involving writing and analytical thinking, which may not be well reflected in introductory level courses.

Conclusions
Overall, findings from our mixed-methods study support the likelihood that the online delivery mode can be as effective in promoting key learning outcomes as traditional face-to-face sociology instruction. We look forward to additional research that employs rigorous controls across delivery modes to replicate whether these same null findings appear in other upper-division sociology classes. Similar studies in related liberal arts courses would also help to provide evidence regarding whether online instruction can effectively build higher order thinking skills including analysis and synthesis. Alternatively, additional studies showing learning differences by delivery mode should prompt instructors to devise ways to render alternative delivery modes equally effective in promoting student learning.
Many of today's faculty have been trained in teaching in face-to-face settings; their teaching experience may be solely in traditional face-to-face classrooms. Some may be uncomfortable with the prospect of online instruction, and given the mixed findings we have identified, such reservations are understandable. However, online instruction is becoming more common in higher education in the 21 st century. We are encouraged that increasing numbers of younger scholars are receiving experience in both face-to-face teaching as well as online instruction, sometimes while still in graduate school. This should help to better prepare them to teach in either delivery mode, or in blended settings. This combination of early exposure to more than one instructional delivery mode and additional research regarding course effectiveness by delivery mode should be a helpful combination for faculty in the years ahead.