Unexpected attitudinal growth in a course combining reformed curricula

In this paper, we show data from the Colorado Learning Attitudes about Science Survey that suggests that Georgetown physics majors become increasingly expert in their attitudes towards physics learning and knowing after taking a course that combines two reformed curricula, Matter and Interactions (M&I) and Tutorials in Introductory Physics (TIPs). This occurs even though the two curricula do not send a consistent epistemological message to students. We analyze interview video data of two of these students to illustrate examples of this growth. We examine video data of one of these students in a tutorial session to describe a possible mechanism that may have contributed to the growth. Finally, we compare this qualitative video data with quantitative data from the newly developed Perceptions of Physics Classes survey and discuss aggregate responses to this survey in considering the ways in which other students developed more expertlike attitudes in this course. We conclude that the attitudinal growth observed cannot be explained simply “as the result of” either M&I or of TIPs but rather find the most plausible explanation to be that the growth is an emergent phenomena produced by M&I and TIPs working together in concert with other factors.


I. INTRODUCTION
Epistemology is a growing area of interest in the field of physics education research (PER) [1][2][3][4][5][6][7][8][9][10][11][12][13]. In this paper, we use the word "epistemology" to mean one's views about the nature of knowledge and knowing [14]. At one level, epistemology is important because funding for scientific research in a democracy is affected by the public's perception of science. At the level of the classroom, a student's view about the nature of knowledge and knowing in physics might affect how she prepares for class and her assessment of how well she "knows" the material being taught. A student who stably sees physics as a plethora of disconnected facts, for example, might be content to memorize equations without considering how they relate to each other or to the real world [15][16][17]. The importance of epistemology on a longer time scale has been studied as well. For example, student responses to various attitudinal surveys administered prior to taking a college physics class have been shown to correlate with which course they choose to take, with conceptual gains within that class [18,19], with student interest in physics [20], and with who decides to become a physics major [19].
A number of survey instruments has been developed to probe student attitudes and beliefs about knowledge and learning in physics. These include the Maryland Physics Expectations (MPEX) survey [10], the Epistemological Beliefs Assessment for Physical Science (EBAPS) [12], the Views About Sciences Survey (VASS) [11], and the Colorado Learning Attitudes about Science Survey (CLASS) [9]. Unfortunately, in introductory physics courses, it is common for students' views about the learning of physics to become less expertlike over the course of the semester [9,10,[21][22][23]. This applies to courses taught with traditional pedagogy as well as to many courses that adopt reform curricula and interactive engagement techniques. The latter courses often focus on improving student conceptual understanding, perhaps with an implicit assumption that epistemological growth would "come along for the ride." Evidence indicates however that this is not a good assumption. There are many cases where, despite strong gains on concept inventories, student epistemologies as measured by attitudinal surveys deteriorate.
Recently, reports of positive shifts in epistemology have begun to appear in the literature [5,[24][25][26][27][28][29]. Since the number of successes is still small, the conditions necessary and sufficient for a physics class to improve student epistemology remain unclear, and there remains a need for theory on how epistemology develops [30]. Many of the courses that have been successful in improving epistemology, such as Modeling Instruction [25,28] and Physics for Everyday Thinking [24], incorporate discussions and activities that explicitly address the nature of scientific knowledge as an integral part of the course. In the Physics by Inquiry curriculum, on the other hand, epistemological gains have been reported even though the curriculum has only an implicit focus on epistemology [27]. Whether explicit or implicit, a common feature of these courses is maintaining the epistemological focus throughout the duration and across all elements of the course. It has been suggested that introducing isolated curricular components to target student epistemology would not likely be sufficient to produce epistemological growth [7]. Furthermore, looking across these success stories, we find that students must engage in activities that encourage them to approach their learning in epistemologically sophisticated ways.
Here we describe a study of epistemological shifts in an introductory course that partners Tutorials in Introductory Physics (TIPs) [31] with Matter & Interactions (M&I) [32]. This course, Physics 151 (P151), is taken primarily by physics majors at Georgetown University (GU). We have previously shown that P151 favorably impacts students' performance on standard mechanics concept inventories as well as on questions that probe their conceptual reasoning abilities at a deeper level [33]. In this paper, we present preliminary (N ¼ 34) CLASS data that suggest that P151 students also grow more expert in their views about the nature of physics knowledge and knowing. We present case studies of two students, Joyce and Felix, 1 to focus on particular areas of epistemological growth. In the case of Joyce, we see an improved attitude about the role of mistakes in learning. In contrast, for Felix, we see an improved attitude about the connectivity of knowledge in physics. These results are particularly significant because, as we will show, TIPs and M&I do not have a common epistemological focus, making our findings an exception to the trend described above.
We briefly consider the mechanism by which this growth occurred. In the case of Felix, we do this by analyzing video data of him working with his classmates during a TIP session. We see the most plausible explanation for his growth is arguably that M&I's emphasis on the unity of physics influences Felix and his group mates to seek coherence and connections in their learning of physics, and they are able to practice doing this in the learning environment afforded by TIPs. These experiences in TIPs of connecting physics knowledge together are similar to the experiences had by students in the epistemologically oriented courses described above. Plausibly, these experiences in TIPs could help P151 students become more sophisticated in their view about the connectivity of physics.
We view the epistemological development of Felix and his classmates not as a direct result of M&I or of TIPs, but rather as an emergent phenomenon that arises from the interaction of TIPs, M&I, and other factors including how the students relate to each other and to their instructors. We view the course as a learning "ecology" [34] in which combining TIPs and M&I results in some expected and some unexpected effects that cannot be described as being "caused" by one curriculum or the other. In filling out our newly developed Perceptions of Physics Classes (Perceptions) survey, Felix and Joyce reported P151 to be strikingly different from their previous physics courses in that it encouraged epistemologically expertlike practices, and these reports are consistent with their accounts given during interviews. Many other students in the course similarly described P151 on the survey as promoting more expertlike views. This suggests that, although the specific mechanism of growth likely varied from student to student, the reformed curricula used in P151 may have played an important role for students other than just Felix and Joyce.

A. Course description and student population
Physics 151 is an introductory calculus-based mechanics course taken primarily by GU students who have expressed interest in majoring in physics. The enrollment is typically between 20 and 30 students. Most of the students are freshmen who have completed a traditional physics course in high school, often at the AP level. These students arrive at GU with a strong background in Newtonian mechanics, as evidenced by strong precourse scores on mechanics concept inventories such as the Force Concept Inventory (FCI) [35] administered in 2009 and 2010 and the Force and Motion Conceptual Evaluation (FMCE) [36] administered in 2011.
The course meets for three one-hour lecture periods, one two-hour laboratory session, and a one-hour recitation session each week. A. Y. L. has consistently been the lead instructor of the course since 2008. Since 2009, the course has used the Matter and Interactions curriculum for the lecture and lab components, and Tutorials in Introductory Physics for recitations. To the best of our knowledge, this is the only course to institutionalize this combination of curricula.
Matter and Interactions is a research-based curriculum that restructures the traditional introductory course to emphasize the intellectual coherence of physics. A theme that runs through the first semester is that only a small number of fundamental principles are needed to explain a wide range of physical phenomena. In addition, M&I introduces microscopic models of matter that provide mechanistic accounts for macroscopic phenomena. For example, the ball-and-spring model for solids is used to explain, among other things, elastic moduli and heat capacity of solids. By connecting topics through fundamental principles and building models that apply to seemingly very different phenomena, M&I aims to show the connectivity of knowledge in physics [32]. M&I can be used in any class format, from a traditional lecture-based course to a SCALE-UP [37] environment, and its effectiveness has been assessed in various class formats at a variety of institutions [38,39]. The GU implementation combines 1 All names are pseudonyms.
interactive M&I-based lectures that use Peer Instructiontype clicker questions with M&I-based laboratory activities that include computational modeling and simple experiments that emphasize concepts.
Tutorials in Introductory Physics is a research-validated set of instructional materials designed to be used as the recitation component in a standard introductory physics course. In TIPs, students work in small groups to complete guided worksheets, doing hands-on activities in some cases and having access to a shared writing space (such as a large sheet of paper in the middle of their table). One of the tactics used by TIPs to address common student conceptual difficulties is an "elicit-confront-resolve" approach. In this approach, students consider questions that may elicit common incorrect conceptual ideas and then are led to a situation in which their ideas are shown to be incorrect or insufficient. They are then guided to "resolve" the conflict in favor of the correct idea [40]. TIPs has been shown to be effective in improving conceptual understanding across a wide variety of topics at different levels [41][42][43][44][45][46][47] with different populations of students [48,49].
On the surface, M&I and TIPs may not seem like obvious partners. First, the sequence of topics in M&I differs significantly from that in the traditional courses for which TIPs were designed. To address this, P151 uses a modified version of the tutorials that aligns more closely with the M&I sequencing of topics [33]. In terms of epistemology, the M&I curriculum emphasizes the coherence of physics through shared, core principles. Although TIPs does not explicitly oppose this idea, and some tutorials are even designed to be used as a sequence of two or three consecutive tutorials, TIPs was not designed with the goal of helping students see physics knowledge as being connected, and there is little in the tutorials themselves that we would expect to serve this role. An instructor interested in helping students develop a more expert view about the connectedness of physics might hesitate to implement recitation activities that do not explicitly reinforce this idea, and thus might hesitate to implement TIPs. Our data, however, suggest that the two curricular elements can work together productively to encourage epistemologically rich experiences. This is significant because, as described in the introduction, most courses that have reported gains on attitudinal surveys have had a consistent epistemological focus across all elements of the course.

B. Previous research findings
In previous work [33], we showed that P151 students achieved normalized gains on the FCI and FMCE that are typical for courses that use interactive engagement methods (FCI pre/post: 76%=87%; FMCE pre/post: 61%=77%). Their scores on the Matter and Interactions Energy Test (MIET [50], administered 2009, 2010, and 2011) also increased, from 39% presemester to 76% postsemester. For comparison, Caballero et al. found that gains of M&I students on the FCI are typically lower than those of equivalent students in a traditional course [51].
One might think that students who have a strong interest and background in physics, like many in P151, will learn physics effectively "no matter what you throw at them." However, this is not necessarily the case [33]. By examining the results of tutorial pretests, we previously identified significant gaps in student understanding even after the topics had been covered in P151 lectures and, in some cases, on homework assignments. Post-tutorial assessments indicated TIPs were effective in scaffolding the development of students' reasoning skills and helping them achieve functional understanding of the material.
One of the pretests analyzed in our prior work was for the tutorial "Motion in two dimensions." Since the specific findings will be relevant for the video data analyzed in this paper, we now review the prior findings for this particular pretest. The pretest poses a well-established question from the PER literature testing student ability to identify the direction of acceleration if given information about trajectory [52]. For a car traveling with constant speed around an oval track, students are asked to sketch vectors indicating the direction of the car's acceleration at various points on the track. The correct answer is that the car's acceleration is perpendicular to its velocity at every point on the trajectory.
The overall performance of P151 students on this pretest was similar to the results reported in the literature for this tutorial [52,53], with almost 20% (N ¼ 65) of students giving a correct response on the pretest, and about 80% being able to draw correct acceleration vectors on the posttutorial assessment. The detailed pretest responses, however, were very different from the findings reported in the literature. More than 40% of P151 students (N ¼ 65) responded that the acceleration was always directed toward the "center" of the oval. This is notably higher than in a traditional course before or after instruction on 2-D kinematics (15%, N ∼ 6900) [52]. Shaffer and McDermott suggested that this response may indicate an incorrect generalization of the case of uniform circular motion, where an object travels at constant speed around a circle.
Prior to completing this tutorial pretest, P151 students had seen examples in lecture showing how the direction of acceleration can be found by drawing velocity vectors and finding the change in velocity. Although it is possible that students who answered that the acceleration points to the center of the oval were struggling entirely because of conceptual difficulties with relating acceleration and velocity vectors, we wondered if they were at least in part hampered by an epistemological predisposition to approach problems by drawing on previously memorized bits of knowledge (in this case, that acceleration for motion on a curved path is "towards the center"). Furthermore, we also considered that the improvement in student performance on the question after tutorial instruction might in part be the result of epistemological development. After analyzing video data taken from class and from student interviews in addition to written survey data, we are now seeing this conceptual and epistemological improvement as an unexpected emergence from the interaction of M&I, TIPs, and other factors including the relationship that the students had with each other.

III. EVIDENCE OF EPISTEMOLOGICAL GROWTH
In this section, we make the case that students in P151 grew epistemologically. We present three pieces of evidence for this growth: First, we describe changes in CLASS score from the beginning to the end of P151 for two different semesters of the course. We also present case studies of two students who were interviewed once during the fall 2013 semester and then again after P151 had concluded. These two case studies serve as illustrative examples of what epistemological growth of students in P151 can look like.

A. CLASS gains
The CLASS was administered to a total of 34 consented students both pre-and postsemester in fall 2013 (N ¼ 17) and fall 2014 (N ¼ 17). On average, P151 students had positive shifts on the CLASS overall score, from 71.2% favorable at the beginning of the semester to 76.3% favorable at the end. As shown in Table I, CLASS scores improved in both semesters; however, in one of the semesters taken alone, the improvements did not rise to the level of statistical significance. Taken together, however, the improvements in CLASS scores are statistically significant at the p < 0.05 level as determined by the twotailed t-test for paired samples. Of the 34 students, 22 had higher CLASS overall scores after P151, 10 had lower scores, and 2 had the same score. Keeping in mind that the sample size is small, these preliminary results nevertheless contrast with reports in the literature showing that student attitudinal scores typically go down after a semester of introductory physics. At best, previous studies using TIPs have reported that student scores did not change from the beginning to the end of the semester [53]. Regarding M&I, to the best of our knowledge, only one study with CLASS data has been published. That study, which was not able to match pre-and postcourse data, concluded that "the CLASS score for those who took the post-test at a minimum did not go down and may even have gone up." [54] For P151 students, the CLASS category with the largest gain was "Applied Conceptual Understanding," which changed from 55.3% to 66.7% favorable responses (p < 0.05). The CLASS statement that registered the largest change, with a 29% shift toward more expert responses, was "When I solve a physics problem, I locate an equation that uses the variables given in the problem and plug in the values." Other statements with large shifts include "Understanding physics basically means being able to recall something you've read or been shown" (20% shift toward more expert responses) and "Knowledge in physics consists of many disconnected topics" (18% shift toward more expert responses). Given the M&I emphasis on reasoning from fundamental principles and on the connectedness of physics, the gains on these questions are perhaps not surprising. Yet the interview and tutorial data presented in Sec. III provide evidence that P151 students are not simply parroting these ideas promoted by the text and lectures. At least some of the students are actually making use of them in their approach to learning.

B. Illustrations of epistemological improvement from interviews
The epistemological growth indicated by the improved CLASS scores for the class overall were also evident in responses that students gave in interviews conducted both at the beginning of the fall 2013 semester and after that semester had concluded. In this section, we describe the interview procedures and present case studies of two students whose interview responses provide evidence for epistemological growth along two different dimensions.

Interview methods
At the beginning of the fall 2013 semester, the 23 students in P151 were invited to participate in hour-long, paid interviews. In total, seven students were interviewed by one of us (M. M. H.) during the first three weeks of the fall semester. The following semester, the 23 students were invited to participate in a post-P151 interview. Of the students who responded, three (Felix, Mel, and Joyce) had been interviewed in the fall. We prioritized interviewing these three students, so as to explore changes in epistemology after a semester of P151. Because of scheduling conflicts, the other respondents (who had not previously volunteered) were not interviewed. Of the three students who were interviewed twice, both Felix and Joyce demonstrated epistemological growth in their interviews, and so we use their interview data as illustrations of what student growth as measured by the CLASS could look like for P151 students. Felix's story is considerably more complicated than Joyce's, and so we spend the most time discussing Felix in this paper. We use Mel's interview data in section IV. D. 5 titled "Results of the Perceptions Survey" to help assess the validity of the Perceptions of Physics Classes survey we constructed. We designed the semistructured interviews to probe student attitude towards physics knowledge and learning. To that end, interview prompts included questions about physics class experiences in both high school and P151. The protocols also contained physics problems that were collected primarily from PER literature [1,15,52,55,56] and organized based upon epistemological themes that emerged from interviews conducted with previous P151 students. If a fall 2013 interviewee touched on one of these themes, M. M. H. would consider asking him to solve some or all of the problems associated with that theme. These problems were each individually printed on paper, with the rest of the sheet blank. Students had access to a pen and scratch paper.
Interviewees were asked to speak aloud as they thought through problems and to not worry about whether their answers were correct. The interview protocol also probed students' epistemological stances with prompts such as "How do you know when you really understand an equation?" [55,56]. The post-P151 interviews were similar to the interviews that took place during P151 but with questions specifically designed for the interviewee based upon the first interview and that student's survey responses. All interviews were audio and video recorded. 2

Joyce
Joyce made several comments in her two interviews which, when taken together, are indicative of a shift in her epistemology. Moreover, Joyce's overall favorable CLASS score was 58% at the beginning of the semester and 78% at the end of the semester, suggesting a significant shift in her views on learning physics. The interview data also provides insight on some of the factors contributing to Joyce's epistemological development.
At the beginning of the first interview, Joyce explained that although she is not a physics major, she enrolled in P151 because physics in high school was a "very challenging class" and she wanted to give the subject "another chance." When asked to elaborate, she said that she had, in fact, "hated" her high school physics, with the following explanation: Joyce: It's very frustrating to like, not know where to go, and then try to solve it and then get the wrong answer. Or, even worse, when you're, when you solve it and you think you know what you're doing and then you get the wrong answer, and like, (laughs) Yeah, that's a little discouraging.
When solving problems and discussing physics principles during the first interview, Joyce would often try to visualize the physical phenomenon. However, she also commented several times that she had had many negative experiences where such an approach led to wrong answers. She explained that Joyce: … the problem is that connecting it to real life has gotten me in trouble a lot because I don't, I don't know how to think about it correctly. Just because, like, when you just imagine things that you observe, um, you can be wrong.
In the second interview, her hesitancy to connect physics to observations, which she demonstrated several times in the first interview, was notably absent. In addition to that, her distaste for getting incorrect answers, which she described in the first interview as being at the root of her dislike for high school physics, had been replaced by an appreciation for the importance of making mistakes in the learning process.
In discussing Physics 152, the electromagnetism course that follows P151 which she was taking at the time of the second interview, Joyce complained that many of the snow days that semester had resulted in TIPs being canceled and that she "missed tutorials." M. M. H. followed up by asking her what exactly she liked about TIPs, and she replied "Tutorial is just a place where you can mess up without any pressure." Joyce: For the tutorial, we would sometimes do things one way and then we'd find out we were wrong, and then we would learn to do the right way… Interviewer: Did you find that to be productive in tutorial, like, doing it first the wrong way and then doing it the right way… I mean Joyce: Yeah, I do… obviously if I have done it wrong before, I can definitely do it wrong in another instance. So knowing that, um, testing a common misconception is usually a good idea for me.
In the case of Joyce, one facet of her epistemological growth appears to be an improved attitude about the role of making mistakes in learning physics. Joyce's specific CLASS responses provide additional data that is consistent with this claim. First, on the item "I do not spend more than five minutes stuck on a physics problem before giving up or seeking help from someone else," Joyce's response changed from "Agree" at the start of the semester to "Strongly Disagree" at the end. This is consistent with the recognition that there is value in persisting in physics, despite not getting the correct answer right away. More generally, the Problem Solving Sophistication category, which includes items like "I can usually figure out a way to solve physics problems," saw a large shift of 33% toward more expertlike views from presemester to postsemester.

Felix
The comments Felix made during the two interviews, as well as his approaches to solving physics problems given during the interviews, are consistent with a large shift towards a more expertlike physics epistemology over the course of P151. Felix's overall favorable CLASS score was 75% at the beginning of the semester and 89% at the end of the semester, providing additional evidence for a significant shift in epistemology. In addition to helping establish that Felix's epistemology became more expert, this interview data also provides insight on some of the factors that may have contributed to his epistemological development.
(1) Felix's first interview. In the first interview, Felix was asked to consider out loud the oval track problem discussed above, where the car is moving with constant speed. His written work is shown in Fig. 1.
Felix: Well, since it's in a circle, I think of … "centerseeking." That would be my guess, but it's not a circular surface, it's an oval. So that's getting me a little bit… But at least, I think, on the turns-not the straightawayslike, C would be a straightaway-it would be pointing towards the center. {Felix draws arrows on the curved ends}… I'll just put an X there … and for C, I think, it's moving more in a straight path, so it would be … maybe it would be no acceleration because it's a constant speed.
Felix's overall solution that "if it's curving, then the acceleration is towards the center of the oval" is incorrect. Furthermore, his initial comment about circles and "centerseeking" acceleration suggests that there is an element of rote memorization in his approach. On the other hand, Felix recognized that the track is similar to but not the same as a circular track, and that the acceleration depends upon the local curvature of the track, and both of these observations are correct and potentially relevant for finding the correct solutions.
Earlier in the interview, Felix said that his approach to studying for exams in high school was to make flashcards. Towards the end of the interview, when asked, he gave an example of the kind of flashcard that he might have prepared for this kind of problem: Felix: Like, maybe like, a circle diagram with where the force is pointing, the velocity is pointing, and the acceleration is pointing, and the centripetal force equation, the centripetal acceleration equation, stuff like that.
Such an approach to studying is consistent with a view about physics as consisting of discrete topics, with facts and equations to memorize for each topic. Felix's solution to the oval track problem in the interview, where he seems to rely heavily on memorized results from the topic of uniform circular motion, could in part be the product of such an approach.
The day after Felix's first interview, he was posed the same question, this time on the tutorial pretest during lecture. Interestingly, on the pretest, Felix said that the acceleration is zero at point C and, at all other points, normal to the track. In Fig. 2, we can see that he reasoned this by drawing what he later referred to as "kissing circles" during the post-P151 interview. Since neither kissing circles nor acceleration in two dimensions was discussed in lecture that day, it is possible that Felix was unsatisfied with his answer to the oval track problem in the interview and independently reviewed or researched the material before class the next day. (2) Felix's second interview. Felix was interviewed a second time a few months after P151 had ended. During this interview, he was asked again about the direction of acceleration for an object moving with constant speed on a curved path, in this case a cycloid. His written work is in Fig. 3.
Felix: For each part of the curve, they have, since it's turning, it has a center-seeking acceleration. An acceleration tangent to its path, an acceleration going along its path. {Felix draws an arrow normal and an arrow tangent to the path at point A} But since it's at a constant speed, this wouldn't be here. {Felix crosses out the arrow tangent to the path and sits for a while silently} MMH: Sorry, what are you thinking… right now? Felix: Um… that since it's at a constant speed, all the acceleration would just be acceleration associated with its changing direction. So they all should be [normal]… They all should be… Hm. MMH: OK, so, I feel like right now at this moment, you have one idea, which is saying that the acceleration is all due to the changing direction, and so it should be [normal], but I feel like there's another idea that's kind of conflicting with that that you're kind of thinking as well. What's the other idea? Felix: The other idea is just pointing to the center of the circle. … Like you know how you would have like a kissing circle or whatever {Felix draws a kissing circle}, or actually, that wouldn't change it, because the center of that would be [normal].
Because the cycloid problem tests the same conceptual reasoning as the oval track problem that Felix solved during the first interview, we can compare the two solutions to look for any improvement. First, the solution to the cycloid problem is correct, whereas the solution to the oval track problem during the first interview was incorrect. Similarities can nevertheless be found between the two solutions. In both cases, Felix noted that acceleration is zero only if both the speed and direction are constant. Also, in both cases, we find the same underlying notion of the acceleration being towards a "center." In the case of the first interview, it was toward the center of the track, and in the second interview, it was toward the center of a "kissing circle." Furthermore, we argue that Felix's solution to the oval track problem-"if it's turning, acceleration is towards the center of the circuit"-is more novicelike epistemologically than his full solution to the cycloid problem: "if it's turning, acceleration has a component in the direction of the path (that is zero if the speed is constant) and a component normal to the path (which changes the direction of motion)." Whereas, one can imagine the former approach as being in part a product of a view towards physics knowledge as consisting of isolated topics, the latter solution is more expertlike in that it uses a more generally applicable approach that one might expect to be produced by a student who sees knowledge in physics as connected.
In the second interview, Felix was asked to compare P151 with his high school physics class. Felix said that, in contrast to P151 where "stuff built off other stuff," his high school class would Felix: Just finish one topic; start another. The order didn't really make sense. Sometimes switch back and forth between subjects. Didn't really connect it to anything else we previously learned.
Regarding P151, he elaborated that "subjects weren't necessarily that related, but Professor Liu connected them so we saw, we were able to connect the dots." Felix was then asked if he had a preference between the two styles and he answered that he "definitely [preferred] having it connected." He added that "there has to be connections because all of this stuff is pretty much related" and then gave the following example: Felix: Well, you know, like, change in values … change in, like, kinematic values, velocity, stuff like that, change kinetic energy, which has to do with, which is connected to work, which is connected to other stuff.
To paraphrase, unlike in high school, P151 showed students how to connect topics together. Now, after the semester has concluded, Felix is viewing physics knowledge as being connected. This is consistent with his selection of "Disagree" to the post-test CLASS prompt of "Knowledge in physics consists of many disconnected topics." At the beginning of the semester, he selected "Agree" for this statement.

Summary
In this section of the paper, we have shown that students left P151 more expert in their views about physics knowledge and knowing than when they entered. This is evident first from the aggregate CLASS data collected for two sections of the course in different semesters. The two sets of student interviews provided more detailed illustrations of this growth: Joyce developed a more expert view about the role of making mistakes in learning, while Felix came to see knowledge in physics as being connected.

IV. POSSIBLE MECHANISMS OF GROWTH
In the previous section, we demonstrated that students in P151 grew in their views about the nature of physics knowledge and knowing. In this section, we will present data relevant for understanding the mechanism by which these students grew. First, we will focus on Felix, looking again at his second interview, as well as his performance solving a problem during a TIP. From this data, we will show that TIPs provided Felix an opportunity to practice connecting old physics knowledge to new physics knowledge and that Felix recognized that TIPs were helpful for his complete understanding of the material. We will further show, however, that the story cannot be as simple as "TIPs improved Felix's epistemology," and we will describe a more plausible mechanism by which his epistemology improved.
After focusing on Felix to show that the reformed curriculum of P151 was an important piece of his story of growth, we will briefly describe how the same can be said for Joyce but in different ways. Finally, we will look at aggregate student responses to the newly developed Perceptions survey, which demonstrates that other students also found P151 to be epistemologically different from their prior high school physics courses in important ways, and we will argue that their improved epistemology as measured on the CLASS might similarly be in part the result of the curriculum used.

A. Felix described TIPs as helping his conceptual understanding
A part of Felix's second interview helps identify factors that may have helped Felix become more expertlike in his epistemology. Early in the interview, Felix was asked how he would describe P151 to a fellow student.
Felix: It (P151) gets a good foundation for understanding physics in the future. It goes on, just basic principles and keeps stressing those principles and having at it until you have it lodged in your brain.
He goes on to credit the tutorials with helping him really understand the material.
Felix: But, like, a major problem I had with the class is we always seemed like we were catching up on what we were learning before. Like, the thing we did… Our tutorials, like, tutorials is when I feel like I really get a grasp of it. But I always felt like that was a week behind everything. Like, we would do our homework on a subject, and I was like, oh, that's fine. I did it, I think I got it. But I didn't really get it until we did a tutorial on that subject like a week later. So like, I did the homework, and I understood it, but I feel like, I got a deeper understanding from the tutorial and I thought it would be a lot better if I did the tutorial before the homework.
When asked to elaborate on the tutorials, Felix mentioned several aspects that he found beneficial.
Felix: You had the thing in front of you, and it was stepby-step, and sometimes there were students that were like … The part where you had three students and you see which one's right and which one's wrong … Those were very helpful. It was very helpful that you had, like, such, the TA and the teacher going around and asking if you needed any help, if you had any questions, trying to explain anything.
Regarding data authenticity, there is little reason to doubt Felix when he says that TIPs is where he "really got it." First, this exchange was initiated by a prompt from the interviewer that was about P151 in general. It was Felix who first mentioned TIPs. Second, Felix's praise of TIPs is in service to a complaint that he has about the course-the tutorials should have preceded the homework, so that he could have understood what he was doing on the homework better. Most importantly, Felix was able to go beyond just saying that he liked TIPs-he was able to back up his claim with examples about what, exactly, he found helpful in TIPs.
Felix said that in TIPs he would "get a deeper understanding." In the data that follows, we will see Felix in a TIP, successfully improving his conceptual understanding about motion in two dimensions. Presumably, this would be an illustration of what he means by "really get[ting] a grasp of it." We will see that he approached his learning in this TIP with the stance that knowledge in physics should be connected. As discussed in the Introduction, courses in which students became more sophisticated epistemologically all had students engaging in activities that encourage them to approach their learning in epistemologically sophisticated ways. It thus seems likely that Felix engaging in TIPs as he did (with the stance that knowledge should be connected) likely helped improve his epistemology as measured on the CLASS. At the same time, however, we will show that his stance existed and persisted without any prompting from the TIPs worksheets and facilitators, showing that there must be more than just the TIPs playing a role in his growth.

B. Tutorial session on 2D motion
In this section, we present video data of Felix working with classmates in the second tutorial, "Motion in two dimensions," which took place shortly after the first interview. The episode presented here features Felix's group wrestling with the direction of the acceleration for an object traveling around an oval track at constant speed (the same question as the tutorial pretest). The intention of this segment of the tutorial is to have students realize that the angle between the velocity at point A and the change in velocity from A to B approaches 90°in the limit that B approaches A. The tutorial then asks students to identify the direction of the acceleration at each point on the track.
In the data that follows, we will see that Felix and his group members attempted to reconcile the new physics knowledge (that the acceleration at each point is 90°to the velocity at that point) with their prior physics knowledge (that acceleration is towards the center of a circular track). Crucial to our argument, we will see that students did this even though they were not prompted by the tutorial worksheets or facilitators to do so.
Twenty minutes into the class, Felix and his group members were considering a question about the limiting value (as point B approaches point A) of the angle between the velocity and change-in-velocity vectors. Felix showed his group members a solution by using two pens to represent the velocity vectors and making them become more and more aligned with each other as point B comes closer to point A. However, Felix incorrectly concluded that the limiting value is 180°.
Often, the students in Felix's group would interrupt each other. To indicate the place where a statement was interrupted, "//" is used. A "/" indicates the start of the interrupting statement [57]. In this exchange between the instructor and Felix's group, we see that the instructor began by asking a question that is similar to what is on the tutorial worksheet. When Felix and S4 replied with an incorrect answer, the instructor proceeded to demonstrate the correct approach and ask them again what the answer was. Such an exchange was not unusual during this tutorial. Both the instructor and the undergraduate teaching assistant (UTA) who was co-facilitating the session generally interacted with students in a way that suggested more interest in the answers to the questions than in student reasoning. In this environment, we might not expect students to be inclined to construct bridges between the new material they were learning and their prior physics knowledge. In the next bit of transcript, we see that the students did this anyway.
Soon after the previous exchange with the instructor, the students began to consider the direction of the instantaneous acceleration vector at point A. Felix: So wait, would it be of that, or would it be, like, whatever arc this is {Felix points to the shared paper} 17 S3: Yeah, that's true… I think it would still be to the center but these would just be longer and these would be shorter.
18 S1: Yeah, I like that 19 Felix: Oh, cool The first answer to this question was given by S4, who wrote on his worksheet that the acceleration is towards the center of the oval. He wrote this without first discussing the issue with his group mates, suggesting that he had little doubt of this response. S3, though finding that answer persuasive, was not as confident, as evidenced by his seeking confirmation in turn 13. Felix was also not sure that that was the correct answer. As he did in the interview, he pointed out that it is not quite the same as a circle, and so maybe the rules are somewhat different. Furthermore, he attended to the local curvature of the path in turn 16. However, S3 was able to find a compromise between the ideas of "towards the center of the oval" and "it depends on the arc" by suggesting in turn 17 that the length of the center-seeking arrows would depend on where on the oval the object is located. S1 and Felix were both satisfied by this. Thirty seconds later, however, S3 again asked S4 about the direction of the acceleration.  Felix: I mentioned that same thing earlier. It's like, yeah, it's, at a curve, so would the circle be, whatever circle that curve, that arc made, so would it points to that center, or the center of the entire oval? 28 S3: Let me look at the book {S3 pulls out his textbook, flips to a particular page, and looks at it for a few moments while the others continue talking} 29 S4: Well based on the fact that 90 degrees is not actually towards the circle … it's just perpendicular to the tangent at a point … 32 Felix: Oh, true, true 33 S1: Yeah, I feel like {S3 puts the book away} no matter where you go on this entire oval, the acceleration vector and the other vector are always at 90 degrees 35 S4: Always equal to 90 degrees … because we already said the limit reaches 90 degrees Shortly after S4 reasserted his confidence about the acceleration being towards the center in turn 21, he did a double-take at about the same time that S3 explicated a dilemma: the idea of the acceleration being towards the center of the oval contradicts what they had worked out (and the instructor had confirmed) before. How can the acceleration be towards the center of the oval if the velocity and acceleration vectors are 90°to each other?
This segment shows the students following the "elicit, confront, resolve" process to recognize that the "towards the center" idea that they might remember from previous physics classes cannot be correct for the oval track. On top of that, Felix introduced his kissing circle idea in turn 27. Although it was not taken up by the others, we see this as an attempt to connect the idea of "acceleration towards the center" to the new material being discussed, not simply replacing the old idea with the new. Note that this idea of kissing circles was not provided by the tutorial, nor did the tutorial give any explanation for "normal to the path" that is also consistent with "towards the center." 4 The fact that Felix thought this is a relevant idea to bring up demonstrates an inclination to make those connections above and beyond what the tutorial encourages. In the next segment, which happened 30 seconds later, other students also displayed this drive to connect physics knowledge. The segment begins with the students calling over the UTA to check their answer, as instructed by the worksheet.  S4 began the interaction in turn 42 by expressing uncertainty about the work they had just done. Notably, this uncertainty persisted even after the UTA checked their answers and confirmed that they were correct and could keep going. This discomfort likely came because S4 had not reconciled his idea of "It should be directed towards the center of the oval for it to move around the oval" (turn 26) with the tutorial-guided conclusion in turn 29 about the acceleration being "perpendicular." In turn 57, S4 directly asked the UTA to reconcile a similar conflict: as the Earth orbits the sun in an elliptical path, the force on the Earth (and the Earth's acceleration) is always directed towards the sun, which is not at 90 degrees to the velocity. S4, we argue, asked the TA his questions because, on some level, he was viewing the interaction with his classmates and with the TA as one in which it is appropriate to make connections between the new material and what he already knew. His sense of not understanding arises because he was thinking that the new material should connect with material he had already learned, and it did not.
S3, like S4, felt sufficiently uncomfortable with the idea of the acceleration being normal to the trajectory to ask the group, in the presence of the UTA, one last time about the direction of the acceleration in turn 50. Although Felix did not ask the UTA any questions, there is still evidence that he was supporting the pursuit of connecting new knowledge to existing knowledge. Primarily, Felix played a role in answering both S3's question in turn 50 and S4's question in turn 59. Presumably, if Felix did not think that those questions were worth asking or that the consequent discussions were worth having, he would not have contributed to those conversations. Instead, he might have remained silent to allow the UTA to more quickly convince his group mates that the answer they had come to is, in fact, correct, so they could move on. We see his utterances, then, as supporting the idea that new knowledge should connect with old knowledge [57,58]. Similarly, even when he was not speaking, his gaze was mostly on S4 and UTA, indicating interest in the discussion. This is meaningful, considering that sometimes a student in TIPs will, for example, continue reading the worksheet on his own while a group mate discusses something with the TA or instructor.

C. Summary of Felix's interview and TIPs data
The interview and tutorial data presented above provide evidence that Felix (and some of his classmates) engaged in the process of connecting their prior physics knowledge with the new knowledge they acquired during P151. On the topic of constant-speed motion in two dimensions, the prior knowledge, which Felix initially tried to apply during the first interview, is that acceleration for an object on a curved trajectory is "towards the center." The new knowledge, developed during the tutorial, is that the acceleration is "normal to the path." First, it is important to note that Felix did not simply replace his prior physics knowledge with a new "rule" introduced by the tutorial. In addition to his "towards the center" idea, Felix articulated during the first interview the idea that the oval is different than a circle and that the acceleration depends to some extent upon the local curvature. He brought both of these ideas up during the tutorial discussion and did not simply let go of them when the "normal to the path" idea emerged. Although the tutorial does not reference kissing circles, Felix invoked them in turn 27 as a way to apply the "towards the center" idea to a noncircular path. Felix's second interview also shows that the "towards the center" idea had not been abandoned. He solved the cycloid problem in two ways, the latter featuring kissing circles, and made the connection that the two approaches are consistent.
In the context of the tutorial, Felix's group demonstrated an inclination to make connections between physics ideas that went above and beyond the affordances provided by the worksheet in an environment that did not particularly encourage this type of coherence seeking. The tutorial facilitators tended to focus more on whether students had the correct answers than on student reasoning. However, the students seemed to not adopt this focus. For instance, even after the UTA confirmed in turn 49 that they had the correct answers, the students continued to ask questions that suggest a reluctance to accept the new knowledge without regard to how it connects with their prior ideas about acceleration being "towards the center." The most direct evidence for our claim that the students were connecting prior and new knowledge is in turn 50: when S3 asks "So does that mean that the acceleration vector is always pointed towards the center of the circle?" S4 replies in a way that makes progress towards connecting the new idea of "normal to the path" with the old idea of "towards the center." He begins in turn 51 by maintaining that he still does think it's true that acceleration is towards the center of a circle, but that an oval is different. At turn 53, S4 says "At these points it's pointing towards the center," probably indicating the four points on the track where this is the case. It would have been sufficient for S4 to have answered S3's question with "no, it's normal to the path, as we just worked through." However, by mentioning the points on the oval in which the acceleration is towards the center, we see S4 attempting to reconcile the "towards the center" idea with the "normal to the path" idea.
Note that the students did not seem to be explicitly cognizant of their inclination to connect physics ideas. They never stated outright that they are uncomfortable about the two ideas not lining up. Evidence that they were uncomfortable is drawn implicitly from interactions such as the one in turns 49-59: Even when UTA granted S4 the confirmation he had originally been requesting, S4's dissatisfaction remained, culminating in his question about the Earth's orbit in turns 57=59.
Despite their efforts during the tutorial session, it seems the students did not fully connect the "towards the center" idea with the "normal to the path" idea. Felix's "kissing circles" solution in turn 27 appears to be an attempt to reconcile the ideas, but it is not clear if even he recognized at the time that the result of that approach will be "normal to the path." Furthermore, the kissing circles idea was not really taken up by Felix's group mates. Had S3 and S4 successfully connected that idea to the "normal to the trajectory" idea, they probably would not have felt the need to seek confirmation for their answers from UTA, since their lack of confidence seemed to come from not knowing what to do with the "towards the center" idea. Nevertheless, the students were able to make some progress towards reaching a better understanding of motion in two dimensions and, more importantly, their method of pursuing that understanding brought them closer to connecting their old knowledge to the new.
Given that Felix and many of his classmates had expressed interest in majoring in physics before taking P151, one might ask whether they came into the class with a sophisticated view about physics knowledge and hence were predisposed to seek connections like they did in the tutorial, even without evident support from the worksheet or instructors to do so. Felix's interviews provide some insight into this. Felix described his prior exposure to physics as a series of disconnected topics in which the order did not even make sense. His strategy for studying relied on flashcards, suggesting a reliance on rote memorization. On the other hand, in P151 Felix said he was shown that it is possible to "connect the dots" between seemingly different topics and ideas. We believe that the inclination of Felix and his group mates to pursue connections between physics ideas in the tutorial arises in part because the M&I-based lectures and text demonstrated to them that such connections exist and that it is important to make those connections.
We summarize the Felix case study by first noting that, as demonstrated between his two interviews (and consistent with his CLASS responses), his views about the connectivity of physics knowledge became more expert. We see this growth as arising in part because of the epistemological message that physics knowledge can be connected that Felix perceived in the M&I-based lectures. Furthermore, we consider the interactions in TIPs such as those we described above as being vital opportunities for Felix to put that message into practice, enabling him to internalize it. Although we do not suggest that M&I and TIPs were the only factors responsible for Felix's epistemological growth, the evidence available suggests that they did play important roles.

D. Mechanism for other students besides Felix:
The Perceptions about Physics Classes survey 1. Joyce As discussed above, Joyce began the fall 2013 semester with a clear distaste for making mistakes. She described it as being "very frustrating" to get the wrong answer. In fact, because of this aversion and because she found that connecting physics knowledge to "real life" often led her to the wrong answer, she reported a conscious effort to stop using her observations in solving physics problems. In her second interview, however, this aversion to being wrong was notably lacking, and, in fact, she described the importance of making mistakes. She also cited the weekly tutorial session as a space in which she felt comfortable making mistakes.
While Felix became more expert in his views about the connectivity of physics, Joyce's interviews suggested that she grew more expert in her attitude about the role of making mistakes in learning physics. Her responses to the CLASS were consistent with this.
Joyce credited TIPs for providing a safe place to "mess up" and for institutionalizing a process of first "do[ing] things one way and then… find[ing] out [they] were wrong." As discussed in the Introduction, a core feature of TIPs is the "elicit, confront, resolve" approach, which Joyce seems to be describing in these quotes. It seems likely that, in the case of Joyce, her improved view about the role of making mistakes in learning physics can be attributed largely to TIPs. The M&I-based lectures and labs may have played very little if any role in this aspect of her epistemological growth.

Motivation for the Perceptions Survey
Case studies like those presented above for Felix and Joyce are useful illustrations to describe in detail the kind of growth that our students' CLASS gains may represent. In addition to these illustrations, however, it is useful to know how many students overall benefited in their epistemological growth from the reformed curricula used. In other words, although the curriculum may have played a role in Felix and Joyce becoming more expert in their views about physics, albeit in different ways, could it be that the other P151 students who raised their scores on the CLASS did so simply because they are physics majors? Generally, we think that is unlikely, particularly in consideration that studies of the epistemological development of physics majors have found that epistemology begins relatively expertlike when students enter as freshmen and does not grow until their senior year [59][60][61]. To address this question with our own students, we created a survey to ascertain whether students perceived a difference in the epistemological focus of P151 in comparison to their previous physics courses. We posit that if, like Felix and Joyce, a student improves his score on the CLASS and reports P151 as having a more expertlike epistemological focus than previous physics courses, then it is likely that the reformed curricula used in P151 played a role in the student's epistemological growth.

Creation of the Perceptions of Physics Classes survey
The Perceptions of Physics Classes survey was developed based on interviews of six students who had taken P151 in the fall 2012 semester. The recruitment and interviewing processes were similar to what was done for the fall 2013 students. Many of the statements made by students who were interviewed in this group indicated an awareness of an epistemological aspect of P151 that had been missing or even undermined in their high school physics classes. To probe the extent to which this was shared by other P151 students, we created a survey. Similar to the creation of the MPEX survey from interview data [10], our "Perceptions" survey was based upon statements interviewees made describing the learning approach encouraged by P151, particularly in comparison to their prior physics courses.
To illustrate, we briefly describe results from the interview with Petra, one of the students interviewed after the fall 2012 semester. In high school, Petra took both an overview course (with both mechanics and electromagnetism) and calculus-based AP Physics C. In the quotes below, Petra explained that she found the connectivity of P151 to be different from what she had previously experienced.
Petra: … we just learned how to do things in high school, so we would look at a circuit and he would say, "okay, resistors in series you add them. In parallel you do the one over,"… [in P151] We went from momentum to everything else we ended up learning. Whereas, in high school it was "this is the equation you're learning today" and you wouldn't really know why those variables, why that worked.
[P151] definitely spent more time trying to teach us why you can go from this equation to another equation. Petra: [In P151] it's "from this topic we can look at this relationship, and then find something new." … [In high school] It was all situational. You had to know you were in that realm, and then you would know how to do it. Because if you're talking about taking a centripetal force test, you know you're going to have to use that equation.
To summarize, Petra perceived that the approach in her high school physics classes was to accept and memorize disparate equations and facts, and that topics seemed to be in discrete "realms." In P151, on the other hand, she felt that students were taught that "you can go from one equation to another" and "everything" was undergirded by fundamental principles, like the momentum principle.
To construct the survey, we first watched three of the interviews (including Petra's) and formed initial characterizations of how each interviewee described P151 in contrast to high school physics and how the interviewee described her approach to physics learning changing as a result of P151. We identified segments of those interviews that were relevant to these issues and selected representative quotes. One of us (M. M. H.) then watched and transcribed the remaining three interviews, and collected quotes that were either similar or contrary to the previously selected ones. Finally, these quotes were paraphrased and grouped with similar quotes. (Does the quote talk about using a plug-andchug approach in a physics class? Does the quote talk about how topics in a physics class are disconnected or connected?) These paraphrased quotes were combined and consolidated into 10 statements, each of which starts with "In physics class,…." The survey asks respondents to indicate the degree to which they agree or disagree with each statement.
For example, related sentiments expressed by Petra and other interviewees were consolidated into the survey prompt: "In physics class, we learned a bunch of different equations and how to use them. We'd take a single equation or a couple of equations and use them to solve a single type of problem. So if it was a question about centripetal forces, you'd know you'd probably need to use that equation." The full Perceptions Survey is provided in Table II below.

Data collection and analysis
At the beginning of the fall 2013 and fall 2014 semesters, P151 students were asked to complete a "survey," which TABLE II. The Perceptions Survey. Students who selected the "expert" response were scored as indicating that their physics course strongly encouraged expertlike behavior.
Prompt given in Likert-scale format "Expert" response 1) In physics class, topics were disconnected. You had Newtonian stuff, how you move it with force and acceleration, then you do potential energy, then springs and then you had your thermodynamics and then you move on. Strongly Agree consisted of the CLASS and the Perceptions survey. At the end of the semester, students were asked to complete another "survey," which included the CLASS, the Perceptions survey, and some questions about their impression of the course. On the presemester survey, students were asked to answer the Perceptions survey based on a previous physics course that, in their mind, was "the most representative of physics class in general" (see Supplemental Material [62]). For all the fall 2013 and fall 2014 respondents, this representative prior course was a high school course. On the postsemester survey, students answered each item on the Perceptions survey for both their representative prior physics course as well as for P151. Both the CLASS and the Perceptions survey consist of Likert-scale prompts, where students choose to "Agree," "Strongly Agree," "Disagree," "Strongly Disagree," or be "Neutral" about a given statement. For analysis of results, CLASS developers recommend condensing the five-point scale to three points, since interviews with students revealed no clear distinction in the minds of students between "Agree" and "Strongly Agree" [9]. In analyzing the Perceptions survey results, however, we did not condense the five-point scale. This is because the postsemester survey asks students to reflect on both their high school course and P151. We wanted to account for the fact that students might distinguish between the two courses by, for example, selecting "Agree" in response to an item for one course and "Strongly Agree" for the other course.
Of the students in the fall 2013 and fall 2014 P151 courses, 34 completed both the pre-and postsemester surveys and gave their consent for their data to be used in research. However, one person of the 34 had never taken a prior physics course and is thus not included in this analysis.

Results of the Perceptions survey
Possible scores on the Perceptions survey range from -20, indicating a student strongly disagreed with each statement describing the class as promoting expertlike behavior and strongly agreed with each statement describing the class as promoting novicelike behavior, to þ20, indicating the opposite. Comparing post-P151 Perceptions responses referring to P151 to pre-P151 Perceptions responses referring to high school physics courses, we find an average increase of 6.4 points, showing that students perceived P151 as encouraging a more expertlike learning approach than their high school physics class (p < 0.01). Comparing Perceptions responses referring to high school physics classes pre-and post-P151, we find that students rated their high school class as promoting a more novicelike approach after completing P151 than they did at the beginning of the semester (by an average of 4.7 points). A possible explanation for this shift is that students were unaware, for example, of the degree to which physics could be presented as unified and connected until after taking P151.
Although almost all of the students perceived P151 to be more encouraging of expertlike behavior than their high school class, the difference varied greatly, ranging from 0 points to 26 points. Post-P151, Felix reported an 18-point difference between P151 and high school on the Perceptions survey. The largest differences were on statements 1, 4, and 8 (see Table II). On statement 4 ("In physics class, new topics drew on what we had learned previ-ously…"), Felix added his own response "Very strongly disagree" for high school, while selecting "Agree" for P151. His survey responses align well with sentiments expressed in his interviews.
Post-P151, Joyce reported a 22-point difference between P151 and high school. Like Felix, Joyce's responses to statements about connecting topics were very different for high school than P151. She also reported large differences on statements about equation understanding and use. Further, in response to the prompt "In physics class, we asked 'why' something works the way it does and emphasized what the concepts mean," Joyce selected "Agree" for P151 and "Strongly Disagree" for her high school class. During her second interview, Joyce made statements consistent with her Perceptions survey responses. For example, she described her high school physics class as not "asking 'why'": Joyce: I actually really like the ball and spring model that this textbook uses because it kind of makes sense using smaller things that are easy to visualize in your head to explain larger phenomenon. But in high school (…) you were just expected to know something without knowing why.
A third student, Mel, in contrast, did not report P151 as being very different from high school physics (1-point difference on the post-P151 survey). There were no statements on the ten-statement Perceptions survey for which she gave a positive response (Agree or Strongly Agree) for her high school physics class but a negative response (Disagree or Strongly Disagree) for P151 (or vice versa). However, in response to the prompt on the second survey that reads "This course was significantly different from my high school physics class(es)," she selected "Strongly Agree." To gain insight into her choice for that question, she was asked in the second interview what the biggest difference between P151 and her previous physics classes was. Mel answered that it was her classmates. In P151, she felt a little overwhelmed by the other students' ability and interest in physics. Eventually, that played an important role in her decision not to take the next course in the sequence. She did not describe any epistemological aspects of P151 in comparison to high school, even when asked if there were differences in the course structure or style of teaching. This is consistent with her failure to identify differences between P151 and her prior physics class on the Perceptions survey.
In summary, of the three students who were interviewed during and post-P151, two perceived a substantial epistemological difference between high school physics and P151 while one did not. In all three cases, responses on the Perceptions survey are consistent with statements made during the interviews, lending credence to the overall survey results for the class.

Discussion
Out of 34 students, 22, Felix and Joyce included, became more expertlike epistemologically as measured by the CLASS and perceived P151 to have a more expert epistemological focus than their high school physics course. The case studies of Joyce and Felix served as illustrations of different types of epistemological growth observed in P151. We do not expect the examples provided above to be exhaustive; we expect that there are others yet to be explored. Nevertheless, the aggregate data suggest that Felix and Joyce were not idiosyncratic in that they (1) developed more expertlike views about the nature of physics and physics learning and (2) were supported in epistemological growth by the reformed curricula used in P151.
We wish to emphasize that we are not claiming that the curricula are the cause of the epistemological growth in the 22 students, or even in Joyce and Felix. Rather, we are saying that, in the case of Joyce and Felix, we find the most reasonable explanation for their growth in light of the fact that the data available involves the curricula playing a crucial role. In the case of Felix, it seems likely that TIPs offered an opportunity to try out the epistemological stance demonstrated in the M&I-based lectures and textbook that physics knowledge is connected. In the case of Joyce, the "elicit, confront, resolve" approach combined with the low-stakes atmosphere of TIPs allowed her to become more comfortable in making mistakes in physics. Although it may play a smaller or larger role than it did for Joyce and Felix, we suspect that the curricula played some role in the epistemological growth of the other 20 students as well.

V. CONCLUSIONS AND IMPLICATIONS
Students in P151, an introductory calculus-based course that uses a combination of Matter and Interactions and Tutorials in Introductory Physics, demonstrate gains on the CLASS attitudinal survey in addition to the previously reported conceptual gains as measured by concept inventories and other measures. These gains are unusual in that student epistemology typically deteriorates after an introductory course, even when substantial gains are measured in conceptual understanding. The gains in P151 are even more unusual because epistemological growth typically occurs in classes that present a consistent epistemological message across all elements of the course. Here, in contrast, the gains emerge in a class in which different curricular components present different epistemological messages. The details of and mechanisms for the gains that were observed on the CLASS were investigated in more detail via a combination of one-on-one student interviews, inclass video, and a new survey, the "Perceptions of Physics Classes" survey, which helps to contrast students' experiences in P151 with prior physics courses.
It is difficult to attribute the epistemological growth that we observed to a single source. Felix, for instance, credited the M&I-based lectures with showing him how physics ideas connect to each other, and he found this markedly different from what happened in his high school physics class. During his second interview, however, he credited TIPs as being vital for his conceptual understanding; he described the TIPs as being the place where he was able to really "get it." During a tutorial, he made progress in conceptually understanding the new material, and he approached this learning with the implicit view that he should connect the new material being learned with physics ideas that he already has. Courses in which students become more epistemologically sophisticated all have students engaged in activities that encourage them to approach their learning in epistemologically sophisticated ways. We thus think that it is at least plausible that Felix's approaching the new knowledge in TIPs with the stance that it should be connected to his old knowledge could contribute to his epistemological development, which, in addition to being qualitatively illustrated in the interviews, was also quantitatively measured with the CLASS. Felix looked to build these bridges despite a lack of epistemological cuing from the worksheet and from the tutorial facilitators. It seems likely that the epistemological message of "physics knowledge is connected" that he picked up in lecture made him more prone to seek these connections in the tutorial.
The nature of the gains and the role played by various curricular elements may not be the same for all students. In contrast to Felix, for example, Joyce's epistemological growth came primarily in her view about the role of mistakes in learning physics. The safe, collaborative learning environment of TIPs and the cuing of the TIPs worksheets seem to have contributed significantly to this aspect of Joyce's growth. For other students, M&I may have played the dominant role. In any case, we view the specific curricular components and their interaction as being only part of the story for how epistemology developed.
It is well known that many factors influence the effectiveness of reformed curricula, including tutorials [63,64]. For example, Madsen et. al. found that class size may be an important factor, with students in smaller courses (like P151) more likely to improve their epistemology than students in larger courses [65]. In the context of P151 TIPs, beyond the worksheets and the facilitators, other factors such as how comfortable students are with their group members and how well prepared the students were both conceptually and epistemologically prior to the class likely played important roles in creating the collaborative learning environment from which Felix and Joyce benefited. What matters ultimately, be it because of small class size, that the students are physics majors, or any other reason, is how the students engage with TIPs, which is determined only in part by the worksheets themselves. Prior research reported negative shifts in epistemology whether TIPs or a traditional recitation was used in conjunction with traditional lectures [10]. In contrast, we have presented here an existence proof showing that, under the conditions necessary to establish and maintain a collaborative space like what was observed in our data, TIPs can contribute to creating epistemologically rich experiences, the likes of which tend to lead to epistemological development.
While the specific findings we report are in the context of the curricular choices and student population of P151, we have more generally presented evidence that curricular components that do not reinforce each other epistemologically can still work together to improve student epistemology. For example, one component of the course impresses upon students that physics ideas can be connected, and this prepares students to work toward finding these connections in other contexts, even under conditions in which there are minimal epistemological scaffolds to encourage such behavior.