Teacher feedback on procedural skills, conceptual understanding, and mathematical practices: A video study in lower secondary mathematics classrooms

(cid:1) We analyzed 47 mathematics teachers' feedback in 172 lessons in Norwegian lower secondary school. (cid:1) Both the feedback quality and its focus (procedures, concepts, or mathematical practices) was charted. (cid:1) Teachers provided clear and speci ﬁ c feedback that largely focused on procedures. (cid:1) Conceptual feedback was also common, but feedback on mathematical practices was rare. Feedback is a prevalent teaching practice in mathematics classrooms, but few studies have documented how mathematics teachers enact feedback in classrooms. We investigated how 47 teachers provided feedback in 172 mathematics lessons in Norwegian lower secondary schools. We analyzed the quality of feedback, the quantity of feedback, and whether the feedback addressed students ’ procedural skills, conceptual understanding, or engagement in mathematical practices. Teachers spent large amounts of time providing concrete and speci ﬁ c feedback, most of it addressing procedural skills while conceptual feedback was less common. The study highlights details of feedback relevant for both pre- and inservice mathematics teacher training.


a b s t r a c t
Feedback is a prevalent teaching practice in mathematics classrooms, but few studies have documented how mathematics teachers enact feedback in classrooms. We investigated how 47 teachers provided feedback in 172 mathematics lessons in Norwegian lower secondary schools. We analyzed the quality of feedback, the quantity of feedback, and whether the feedback addressed students' procedural skills, conceptual understanding, or engagement in mathematical practices. Teachers spent large amounts of time providing concrete and specific feedback, most of it addressing procedural skills while conceptual feedback was less common. The study highlights details of feedback relevant for both pre-and inservice mathematics teacher training.
© 2021 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).
Hundreds of studies have investigated feedback and its effects on student learning, and meta-analyses and reviews indicate that teachers' use of feedback has large impacts on student learning (Hattie & Timperley, 2007;Kluger & Denisi, 1996;Shute, 2008). Feedback is a prevalent teaching practice, as eighty percent of teachers state that they frequently or always "observe students when working on tasks and provide immediate feedback" (OECD, 2019, p. 61). Despite the abundance of studies on feedback and its importance in regular classroom teaching, few studies have investigated feedback provided in classrooms (Ruiz-Primo & Li, 2013;Shavelson, 2003;Wiliam, 2018). This is particularly the case for the mathematics education: research on feedback in mathematics has mostly studied evaluative written feedback (correct/incorrect) on assessments or homework (M. Li et al., 2011) and there are almost no studies of how feedback is provided in regular mathematics lessons (Ruiz-Primo & Li, 2013). The field needs studies of the feedback from in situ and authentic mathematics classrooms because this might give important information on the kinds of mathematical learning opportunities teacher feedback grants students. This can aid mathematics teachers and teacher educators by showing the possible feedback practices available, including best-practice examples, and provide guidance about where to focus teacher development efforts.
The few observational studies of classroom feedback in mathematics lack a mathematical specification, either because they study generic aspects of feedback such as frequency and praise (N. Li et al., 2016;Voerman et al., 2012) or because they conflate mathematics classrooms with other subjects (Gamlem & Munthe, 2014;Voerman et al., 2012). Such studies provide information on possibly very important aspects of feedback, but no information on how teachers' feedback practices support students' development of the mathematical competencies that current mathematics curricula specify. Mathematics educators and curricula stress that mathematical competency includes procedural skills, conceptual understanding, and the ability to engage in mathematical practices such as problem-solving, proving, and mathematical modeling (Ball, 2003;Burkhardt & Schoenfeld, 2019;Hiebert & Grouws, 2007;Ministry of Education and Research, 2015 [UDIR]; Moschkovich, 2013;National Governors Association Center for Best Practices, Council of Chief State School Officers, 2010 [NGACBP]). An important aspect of teacher feedback in mathematics is therefore how it may support the development of these competencies.
In the present study, we investigated teachers' oral feedback to students in 172 video-recorded lessons in 47 mathematics classrooms in lower secondary schools in Norway. We aimed at charting generic and subject-specific indicators of feedback, how much time teachers spend providing feedback, and in what activity formats (individual work/group/whole-class), as well as the mathematical competencies addressed in the feedback. First, to chart generic quality indicators of feedback, we used the standardized observation instrument Protocol for Language Arts Teaching Observation (PLATO; Grossman et al., 2013). Second, to describe how teachers focus on the different mathematical competencies in feedback, we conducted a qualitative analysis of feedback instances in the videorecorded lessons. This also afforded us to chart the amount of time teachers spend in feedback conversations with students and in which activity formats. The key questions investigated were: (1) What is the quality of feedback as measured by the subjectgeneric dimensions? (2) How much time do teachers spend providing oral feedback and in what activity formats? (3) To what extent and how do teachers in mathematics classrooms provide feedback focusing on procedural skills, conceptual understanding, and mathematical practices?

Norwegian context
Norway has a comprehensive and non-tracked model of primary and lower secondary education for the first 10 years of schooling. Mathematics teaching in lower secondary classrooms has traditionally been characterized by teacher-led whole-class instruction and individual work (Bergem, 2014;Olsen, 2013), but more recent studies have seen larger instructional repertoires including group work . Feedback has been strongly emphasized in national policy: the national guidelines for teacher education state that prospective teachers must learn to give productive feedback (Gamlem, 2015;Munthe & Melting, 2016); between 2010 and 2018, a nation-wide professional development program was conducted emphasizing effective feedback in the Assessment for Learning (AfL) framework (UDIR, 2018); and students have a legal right to receive individual feedback (Education Act, 2006, x 3e10). Since 2006, the national curriculum has stressed the teacher's role in familiarizing students with mathematical practices, such as problem solving, mathematical discourse, justification and generalization (UDIR, 2006). The new national curriculum (UDIR, 2015) increases this emphasis by including many of these mathematical practices as "core elements."

The concept of feedback
Feedback is information provided to a student about the student's performance or understanding (Black & Wiliam, 1998). A key feature of feedback is that it is evaluative, either through decomposing qualities of student work or suggesting improvements and ways forward (Sadler, 1989). This understanding is closely related to AfL (see Black & Wiliam, 1998), a set of teaching practices that have strongly impacted educational policy internationally (Birenbaum et al., 2015) and in Norway (UDIR, 2018). Following Black and Wiliam (1998), teachers teach according to AfL if they continually and informally assess student understanding and use this information to adapt their instruction or provide feedback.
In this study, we focus on the oral feedback provided by the teacher in mathematics lessons. Oral feedback can take the form of short praise, encouragement, and corrections, or it may be part of a longer conversation between teacher and student. Questioning plays a key role in providing feedback, because it is often through questions that the teacher learns about student understandings (Small & Lin, 2018). The distinction between teacher questioning and feedback may be blurry. To demarcate which utterances are considered feedback in the present study, we require that the utterances contain a) an evaluation or b) a suggestion for how to move forward. This will generally not include questions, although some questions may fulfill the requirements.

Qualities of feedback
Although aspects of quality feedback varies with the subject taught (Smith & Lipnevich, 2018), researchers have identified several aspects of high quality feedback that are generic (Hattie & Timperley, 2007;Shute, 2008). Feedback should focus on the student's work on the task at hand, and not on the student's skill level in comparison to others or to a grading scheme (Hattie & Timperley, 2007). Feedback is more effective when it not only provides information about answer correctness, but also elaborates on qualities of student work or how to improve (Shute, 2008). These elaborations should be specific and clear (Shute, 2008), but not so specific as to remove student agency from the learning process and making further work too easy (Hattie & Gan, 2011). Rather, they should address the overarching goal of the task and the underlying skills required to perform proficiently (Wiliam, 2018).

Feedback and activity formats
Feedback may be provided in all activity formats, like wholeclass instruction, group work, or individual work. In whole-class instruction, the initiation, response, and evaluation/feedback (IRE/ F) pattern has been used extensively (Inagaki et al., 1999;Mehan, 1979). The IRE/F pattern is an interaction pattern wherein the teacher initiates a question (often termed funneling questions (Wood, 1998)), a student replies, and the teacher gives evaluative feedback. Teacher replies are often short, even when students reply erroneously (Santagata, 2005). Other ways of providing feedback in whole-class instruction exist, for example by discussing students' solution methods (Stein et al., 2008), but they are less common. During group work, the teacher may provide feedback on the students' collaboration or on the quality of the group's work (Brodie, 2000). Individual feedback may have the biggest potential to be R.B. Stovner and K. Klette Teaching and Teacher Education 110 (2022) 103593 tailored to each student's individual needs (Krammer, 2009).

Quantity of feedback
There is no optimal quantity of feedback teachers should strive for. On the one hand, teachers must provide enough feedback that student work is productive. For example, teachers can provide corrective feedback to ensure that students are practicing procedures correctly. On the other hand, feedback may remove important learning opportunities, for example if students are working on challenging tasks and the feedback reveals too much of the solution (Stein et al., 1996). Feedback also comes with "opportunity costs", namely that the time the teacher spends providing feedback might be better spent doing something else (Hays et al., 2010).
Studies from classrooms have found that teachers spend little time providing feedback (Krammer, 2009;N. Li et al., 2016;Voerman et al., 2012). Voerman et al. (2012) found that across 72 lessons (25 in mathematics), teachers provided an average of seven feedback interventions every 10 min of instruction. The authors "find these outcomes to be alarming, because feedback […] is one of the most important tools available to positively influence their students' learning" (Voerman et al., 2012(Voerman et al., , p. 1113). However, we could argue that seven feedback interventions in 10 min is quite a lot of feedback depending on the teaching situation. In the current study, we report on quantity of feedback teachers provide, but do not judge whether too much or too little feedback was provided. Rather, we view the quantity of feedback as one of several parameters that are relevant for understanding how teachers use feedback in the classroom.

The mathematical focus of the feedback
We stated earlier that feedback should elaborate on qualities of student work and what they need to do to improve. A key aspect of this study was to investigate what the mathematical focus of the feedback was. Especially, we were interested in feedback focusing on a) procedural skills, b) conceptual understanding, and c) mathematical practices such as problem-solving, proving, and mathematical modeling. All three areas are underscored as critical in current mathematical curriculum documents in the US and Norway (NGACBP, 2010; UDIR, 2013).

procedural feedback
Procedural feedback focuses on procedural skills by helping students arrive at the correct answer by applying the correct procedure or remembering facts and rules. This conceptualization stems from the classical definition of procedural knowledge, defined as knowledge of isolated facts or rules or rigid procedures (Hiebert & Lefevre, 1986). Examples of such feedback are: pointing out a computational error, showing the next step in a solution method (or even the entire procedure), reminding the student of a rule, and acknowledging whether a student's answer is correct without further elaboration. Verifying whether students' answers or methods are correct aids learning, especially for learners with low background knowledge (Fyfe et al., 2012). Procedural feedback presumably has the greatest effect on procedural outcomes, but some studies have shown improvement in conceptual outcomes as well (Fyfe & Brown, 2017). However, these findings are from experiments conducted individually outside of classrooms, and further research is needed to conclude that the findings are transferable to classroom settings (Booth et al., 2017).

Conceptual feedback
Conceptual feedback is feedback that addresses mathematical concepts, the relations between them, or students' understanding of them. An example is a teacher that draws a graphic representation of measurement division to a student that has made a division error in order to explain why it was an error. Here, only showing the rules of long division would be procedural feedback. Conceptual feedback is a way of explicitly addressing conceptual understanding, which is a critical aid for student learning (Hiebert & Grouws, 2007). As most studies of feedback concern feedback about answer correctness or procedural remediation of errors, not many studies of conceptual feedback exist, and their findings must be further confirmed and elaborated. Although procedural feedback has strongest effect on student learning on similar tasks, conceptual feedback might be more effective on far-transfer tasks where students must apply their knowledge in new contexts (Alibali, 1999). However, providing conceptual feedback is hard: Student teachers instructed in responding to students' concepts and ideas continue to provide feedback through praise or correcting the procedure (Son & Crespo, 2009) even when they acknowledge that the student lacks conceptual understanding (Bartell et al., 2013). Also, it is more difficult to identify student errors and respond to them under time constraints (Pankow et al., 2018). Because teachers must provide feedback "on the spot" during classroom teaching, we expect to find few examples of conceptual feedback in the present study.

Feedback on mathematical practices
Feedback on mathematical practices focuses on how students engage in mathematical practices (Moschkovich, 2015). The term mathematical practices is inspired by the practices mathematicians engage in while they work, such as problem-solving, justifying mathematical claims, or engaging in mathematical discourses (Moschkovich, 2013). A common explanation is that "[there are] two main (and deeply intertwined) aspects when doing mathematics: mathematical content and mathematical practices" (Schoenfeld, 2014, p. 500). Mathematics curricula have increasingly focused on mathematical practices as an important goal for student proficiency (UDIR, 2015;NGACBP, 2010). A problem is that mathematical practices may remain hidden to students if they are not addressed explicitly (Selling, 2016), and that they often devolve into mere prescriptions if a teacher explicitly prompts the use of them (Bieda, 2010). A proposed solution is to link students' work with a mathematical practice while they are engaged in them or immediately after (Selling, 2016), for example through feedback. An example of feedback on mathematical practices is a teacher that gives feedback on a student's use of problem-solving heuristics, for example by evaluating qualities of how the student problemsolved.

Data and design
This study was part of a larger classroom video study conducted to analyze teaching practices in Norwegian language arts and mathematics . During the 2014e2015 school year, the research team video-recorded 172 lessons in grade 8 mathematics classrooms in Norway. The students were 13e14 years old and in the first year of lower secondary school. Forty-seven formally trained mathematics teachers, each teaching in a different school, were video recorded. Three to four consecutive lessons were recorded in each classroom. The schools were carefully sampled to reflect variations in important school variables in Norway, such as rural/urban location, socioeconomic status, immigrant student population, and achievement level on national tests in reading and numeracy. The videotaped lessons followed the classes' normal schedules, and the teachers were asked to teach "as usual". Two discreet cameras and two microphones captured the whole-class discussions, teachers' talk, and teacherestudent interactions (see  for further descriptions of the camera setup). In the 47 recorded classrooms, there were 991 students: 456 boys, 500 girls, and 35 of undisclosed gender. The class sizes varied between 18 and 28 students and averaged 21. The students scored close to the national average on the national numeracy tests when they entered grade 8 and made slightly higher gains during the year.

Analyses
We performed the analyses in two steps. First, we used a standardized observation instrument to answer the research question about the feedback quality in the observed lessons. Second, we conducted a fine-grained qualitative analysis of feedback instances to answer the research questions on (a) the time spent providing feedback, (b) the activity formats in which it was provided, and (c) how teachers provided feedback with different mathematical foci (i.e., procedures, concepts, and mathematical practices).

First step of the analysis: the PLATO observation instrument
We used a standardized observation instrument (Bell et al., 2018) to obtain an overview of the generic quality of feedback across all lessons. Such instruments are suitable for transparently and consistently analyzing large observational data sets . We used the Protocol for Language Arts Teaching Observation (PLATO) instrument  for two reasons. First, its conceptualization of feedback aligns with AfL, which is how teachers in Norway are expected to provide feedback. Second, it captures the generic quality indicators of feedback described previously; namely that high-quality feedback elaborates on qualities of student work or provides substantive suggestions for how to continue. PLATO was initially designed to capture elements of English language arts instruction in middle school but has since been used at the lower secondary level to rate mathematics instruction (Cohen, 2018;Luoto et al., in review;Tengberg et al., 2021). We did not use a mathematics-specific observation instrument, as these do not currently have dimensions that measure feedback (Praetorius & Charalambous, 2018).
The teaching practices that are covered by PLATO correspond with notions of quality mathematics teaching in Norway  and in the US (Cohen, 2018). Specifically, PLATO values teaching that stresses "rigorous content and intellectually challenging tasks, the centrality of classroom discourse in developing sophisticated understanding of content and disciplinary skills, and the critical role of teachers in providing instructional scaffolding for students to help them succeed" (Grossman et al., 2014, p. 295). Feedback within this view of teaching should not just help students by correcting their work or providing steps for how to continue but develop sophisticated understanding of the mathematics and discipline-specific ways of working.
When rating instruction with PLATO, lessons are subdivided into 15-min segments. The segments are rated on 12 elements of quality teaching, like feedback, time management, and quality of explanations. Each element is rated on an ordinal scale from 1 to 4. Only the feedback element was used in the present analysis.
A summary of PLATO's feedback element is provided in Table 1. At the high end, for a segment rating of 3 or 4, the feedback must (i) be clear, (ii) be specific to the student's work or ideas, and (iii) address "substantive elements" in the task or the student's contribution. "Substantive elements" refers to elements that go beyond a specific task. It refers to feedback that addresses students' understanding, underlying skills, or ways of working and is described in contrast to feedback providing prescriptive procedures or rules required for completing a task. For a rating of 3, although the segment may include many vague or procedural feedback instances, it must include at least two substantive feedback instances. For a rating of 4, the segment must include several substantive feedback instances and few procedural ones. Additionally, the rater must be able to reasonably infer that the feedback helps students. There is thus both a qualitative and quantitative difference between ratings 3 and 4. At the low end, for a rating of 1, the teacher does not provide feedback, and for a rating of 2, the feedback is either vague or perfunctory (e.g., "Good job"; "That's correct") or pertains only to correctly performing procedures and completing the task at hand.
Raters must be trained and certified in using PLATO, which requires rating a master-rated collection of 15-min segments with 80% exact agreement. A team of five certified PLATO raters conducted the ratings. All lessons were subdivided in 15-min segments and rated by one rater. The inter-rater reliabilities were estimated by double-coding 16 lessons sampled to include all PLATO rating levels and be taught by different teachers. This constituted 15% of the material since the lessons were slightly longer than average. The feedback element had a 72% agreement between the raters . Upon disagreement, the ratings from the first coding pass were used. We had some concerns about our level of reliability. Reported guidelines for interpreting percentages of exact agreement, propose that 80% is sufficient (Hartmann et al., 2004), but that for more complex and high-inference instruments (such as PLATO), an exact agreement of 70% between raters may suffice (Hill et al., 2012;Ho & Kane, 2013). The reliability level is also briefly discussed in the Results section.

Second step of the analysis: analyzing feedback instances
In the second step, we conducted a fine-grained qualitative analysis using feedback instances as the unit of analysis. We define a feedback instance as a sequence of utterances between the teacher and one or more students that was about their work or ideas and contained a teacher utterance that included an evaluation or a suggestion for improvement. A feedback instance started when a teacher and student began talking about the student's work or ideas and ended when the topic changed, which could be the start of a new feedback instance, or the discussion ended. Feedback instances included short one-line teacher utterances ("Good work.") as well as conversations where the teacher and the students talked and asked questions about the student's work at length. Commonly, the teacher added additional instructiondsometimes quite lengthydto help the student. This was included in the feedback instance only if the teacher tied the instruction very closely to the student's work. Feedback instances could occur during individual or group work or whole-class instruction.
We identified feedback instances using the video-analysis software Interact (Mangold, 2019), which facilitated recording each instance's start and end times and appending labels and comments. Each instance was labeled with the appropriate classroom and 15-min segment, that segment's PLATO feedback rating (1, 2, 3, or 4), the activity format the feedback was provided in (whole-class/group/individual), and its mathematical foci. We also wrote short memos of each feedback instance, which contained a summary of what was said. An example of the information recorded is provided in Table 2. In analyzing the time spent, we calculated the total time a teacher spent in feedback utterances during each 15-min segment.
We then conducted a qualitative analysis of the feedback instances to gain in-depth information about the features and characteristics of how feedback with different mathematical foci was R.B. Stovner and K. Klette Teaching and Teacher Education 110 (2022) 103593 provided across the different PLATO rating levels. We coded features of how the feedback was provided to obtain themes (Braun & Clarke, 2006) of how feedback with the different mathematical foci was provided. Themes were gradually reworked by merging or refining existing themes to better characterize how feedback of different types was provided. Finally, we examined the coding by looking for inconsistencies, checking that the themes cohered across the material, and verifying our interpretations by doublechecking the coding with the videos. We then translated and transcribed the examples that best typified the feedback categories.
In the second step of the analysis, we analyzed a subset of the 15-min segments (n ¼ 54). This subset's size had to be big enough to represent typical patterns in the data set, but small enough to enable a careful in-depth analysis. First, we chose not to analyze segments rated 1 because these contained no feedback. Second, we randomly sampled segments rated 2, 3, and 4 until saturation (Saunders et al., 2018). Saturation was achieved when analyzing an additional 3e4 segments only yielded feedback instances that fit into the existing themes without developing new themes or changing the existing themes' characteristics. The final subset included 54 segments, in which we identified 603 feedback instances. We double-coded 79 feedback instances (13%) that covered all teachers and coding levels. Agreement was high for all codes, including activity format (96%), procedural feedback (90%), conceptual feedback (90%) and feedback on mathematical practices (99%).

Limitations
The study has several limitations in providing valid and reliable observational data on how feedback is provided in mathematics classrooms. We knew that the students in our sample had slightly higher gains on national numeracy tests than the national sample. Our sample might therefore have slightly more high-quality feedback practices than a random sample. Also, the camera's presence may have affected teachers and students, such that the lessons were not typical of normal teaching practices. Empirical research on camera reactivity has found, however, that it fades after the first few minutes and the teaching quality is unaffected (Praetorius et al., 2017), suggesting that camera reactivity was a minor problem. Additionally, the study focused on the teachers' feedback. However, it has been argued that feedback must be seen in relation to other factors, such as the learning goals, the tasks (Small & Lin, 2018), and the individual students (Van der Kleij, 2019). We did not investigate such factors, which would have required a different study design.
Although PLATO was developed to code language arts instruction, it has also been used to code mathematics instruction (Cohen, 2018;Klette et al., 2021;Tengberg et al., 2021) and other subjects such as foreign language and science (Brevik, 2019;Mahan et al., 2021). Current debates on using generic versus subject specific observation instruments to analyze instructional quality provide arguments for both approaches (Berlin & Cohen, 2020;Hill & Grossman, 2013;Schoenfeld, 2018). Praetorius and Charalambous (2018) reviewed 11 generic and subject specific observation instruments for use on mathematics instruction, and concluded that there are benefits and pitfalls of using both approaches. We decided to use PLATO as a generic instrument in the present study and supplement them with a mathematics-specific analysis in step 2.

Distribution of subject-generic quality of feedback
Across the 172 lessons, 520 15-min segments were rated with the PLATO instrument. Of these, 72 (13.9%) were rated 1 indicating no feedback, erroneous feedback, or confusing feedback. These segments showed teachers performing administrative tasks, such as discussing an upcoming school-trip, lecturing, or conducting short tests. In our data, no segment was rated 1 because the feedback was erroneous. There were 349 (67.1%) 15-min segments rated 2, indicating vague, perfunctory feedback or feedback containing prescriptive rules and procedures. In line with the PLATO instrument, the feedback at this level could include one instance with substantive elements, but if two instances occurred in the same segment, it was rated 3. Feedback rated 3 or 4 is considered "high end" according to the PLATO instrument, meaning that they provide specific and substantive suggestions for how to improve students' work or understanding. In total, 91 (17.5%) segments were rated 3; that is, they contained at least two substantive feedback instances that were specific to students' work or ideas. Only eight segments (1.5%) were rated 4, indicating teaching that consistently provided feedback that was both substantive and specific to the Feedback is perfunctory ("well done," "nice work," "correct," "No, wrong.") Prescriptive suggestions for how to continue ("These should be on a common denominator.") Showing correct procedure to complete student's task.
Feedback specific to students' work or contribution. At least two feedback instances contain substantive elements, such as explaining why the student's procedure is wrong, explaining a concept the student has misunderstood, or providing flexible suggestions for how to continue.
Feedback is consistently both specific to student's work or contribution and has substantive elements. It is reasonable to infer that the feedback helps students understand. students' work, and it was reasonable to infer that the feedback helped the students. These ratings were unequally distributed among the 47 teachers. As shown in Fig. 1, the teachers varied widely in the number of high-rated segments they taught. Fourteen teachers (30%) never received a score of 3 or 4; that is, they rarely or never provided substantive feedback. The remaining 33 teachers (70%) received at least one rating of 3 or 4 because they provided substantive feedback during the segments. Five teachers (11%) received at least one rating of 4, meaning that feedback was consistently substantive.
No teachers received high ratings for all their segments, which was neither expected nor deemed beneficial. We regard the PLATO results as a descriptive overview of the teachers' feedback practices, and higher ratings are not always better because many learning situations may benefit from not including feedback (Hays et al., 2010) or because procedural feedback may be sufficient for the learning at stake. Next, we describe how much time teachers spent in feedback instances, in what activity formats feedback was provided, and how teachers provided feedback on procedures, concepts, and mathematical practices.

Looking across feedback instances: activity format and time spent
We conducted the second step on a subsample based on the PLATO ratings as described in the Methods section. The final subsample in step 2 of the analysis contained 54 segments and 29 teachers: 29 segments rated 2, 17 segments rated 3, and all 8 segments rated 4. We identified a total of 603 feedback instances. A feedback instance was a sequence of utterances that was about student work or ideas that included an evaluation or a suggestion for improvement. An overview of how they were distributed according to PLATO feedback rating and activity format (i.e., individual work, group work, or whole-class instruction) is shown in Table 3.
Fewer feedback instances were observed during group work than in individual work and whole-class instruction. We should clarify that during individual work, the students were often seated in pairs or in groups of two, three or four, as is common in Norwegian classrooms. Students may talk about mathematics and ask one another for help, but they are supposed to solve the tasks individually. We labeled the feedback instance as occurring during group work only when it was clear that the assigned tasks were meant to be solved in groups.
The teachers spent a large amount of time providing feedback to students, see Table 4. Around half of the time in the 15-min segments analyzed in step 2 was spent in feedback instances. The average time spent providing feedback did not differ greatly in the segments rated 2, 3, and 4, which indicates that segments have different ratings because of the qualitydnot quantitydof feedback.
The time spent providing feedback occurred mainly during individual work or group work. Here, the teachers provided feedback almost constantly. The only time during individual and group work that teachers were not in feedback instances was when moving between desks or when students asked non-mathematical questions, such as about homework deadlines. A significant amount of time was spent providing feedback during whole-class instruction as well, as teacher lectures without student participation were rare. Rather, teachers asked questions that students answered. Students' answers were usually short, whereas the teachers' feedback was sometimes long. Thus, a considerable amount of time was spent in feedback instances in all activity formats. We now move to the  mathematical content in the feedback instances. Fig. 2 shows the proportion of feedback instances in analysis step 2 that addressed procedures, concepts, and mathematical practices for each PLATO rating level. It shows that higher rated segments have higher proportion of conceptual feedback and feedback on mathematical practices. This is an indication that segments with mostly procedural feedback was rated 2 and segments with substantive feedback was rated higher, despite our concerns about the inter-rater reliability of the PLATO coding. The proportion of procedural feedback is approximately the same at all rating levels. The reason is that when teachers gave conceptual feedback, they almost always addressed procedures as well. Qualitative descriptions of how this was done is given in the next sections.

Characteristics of procedural feedback, conceptual feedback, and feedback on mathematical practices
Only feedback instances with a mathematical focus are shown in Fig. 2, resulting in the columns for rating levels 2 and 3 adding up to less than 100%. The remaining feedback instances contain mostly vague praise ("Good work") and some feedback on effort ("It was hard for me to get you to start working today, class. This will need to improve tomorrow."). This means that most feedback was focused on mathematics, in contrast to earlier research showing that feedback mostly functioned as emotional support (Gamlem & Munthe, 2014).
In the following, we present characteristics of feedback instances that focused on procedures, concepts, and mathematical practices. The presentation is structured according to PLATO rating such that the results from step 2 can elucidate the distribution of PLATO scores in Fig. 1 above.

Feedback rated 2: procedural feedback
In the 29 segments rated 2, we identified 303 feedback instances (see Table 3). A PLATO feedback rating of 2 specifies that the feedback is vague, perfunctory, or only attends to procedural aspects of the task. In the segments rated 2, we distinguished between two categories of feedback instances: showing students correct solution methods and brief evaluation of student answers.
Procedural Feedback: Showing Students Correct Solution Methods. A prominent feature of the feedback in the segments rated 2 was how the teachers helped students by giving clear and unambiguous guidance in completing the task on which the students were working. We call this showing students correct solution methods. It was present in all segments in the subsample analyzed in step 2 and was especially prominent during individual work and group work. Some feedback instances were brief, such as when the teachers only stated what students should do without showing them the complete procedure (e.g., "remember to put on a common denominator here"). However, many feedback instances were longer, wherein the teachers carefully demonstrated how to conduct the procedure. In the following example, the students were simplifying fractions during individual work.
Example 1: In this example, the teacher provided all the steps but activated the student by prompting her to perform the simple calculations. Procedural feedback generally included such student prompting or funneling questions (Wood, 1998) about the next step. Note that the teacher did not include substantive elements, such as relating the explanation to general rules or concepts that could help the student understand the procedure.
We warn against inferring that the instruction in segments rated 2 was "traditional" or "teacher-led" because of the above descriptions. Indeed, the students were active in most segments, and peer-to-peer interaction was the norm regardless of whether the students solved routine textbook exercises or more demanding problems.
Procedural Feedback: Brief Evaluations of Student Answers. In segments rated 2, most feedback was very short, and most feedback instances lasted less than 15 s. In these instances, the teachers mainly acknowledged whether the students' answers were correct, perfunctorily re-voiced the students' answers, and often included brief praise, such as "That's correct, great!" We termed this kind of feedback brief evaluations of student answers. During whole-class discussions, brief evaluations of students' answers accounted for almost all feedback instances. For example, one teacher asked in the beginning of the lesson whether the students knew examples wherein statistics was used. Many students gave examples, and the teacher replied, "Great example," or "Well, I am not so sure about that one." A feature of these brief evaluations of student answers was that the teachers did not dwell on the students' errors but proceeded quickly to the correct answer. The following example shows two feedback instances (lines 2e3 and 4e5) during a discussion in which the teacher explained how to solve the equation xþ1 10 ¼ 2. Example 2: 1 Teacher Any ideas? 2 Student 1 You can divide x by 2. 3 Teacher Well, that is not quite true. We must first find what x is. 4 Student 2 Multiply by ten and subtract one. 5 Teacher Yes! [writes on the board] What do I need to do now? Fig. 2. Feedback instances in segments rated 2, 3, and 4. The percentages of the columns may add up to more than 100% because each feedback instance can be of several types.

R.B. Stovner and K. Klette
Teaching and Teacher Education 110 (2022) 103593 The student's response in line 2 was apparently incorrect, which the teacher clearly stated in line 3. The teacher commented briefly on why it was incorrect ("We must first find what x is"), but it is unclear how this comment related to the student's answer, and the teacher quickly asked a new student to answer. This is typical of how the teachers responded to erroneous responses in procedural feedback instances: it only gives information on whether if a correct step has been executed.

Feedback rated 3 and 4: specific and substantive feedback
In the 17 segments rated 3 and the eight segments rated 4, we identified 219 and 81 feedback instances, respectively. In PLATO, to gain a rating of 4, the feedback must be consistently specific to student work and ideas and attend to substantive elements of the task. A rating of 3 is similar, except that the feedback need not be consistently substantive (but it must have at least two substantive feedback instances). That is, a rating of 3 indicates a mix of procedural and substantive feedback. The below categories thus apply strongly to segments rated 4, whereas segments rated 3 also include elements similar to segments rated 2. We distinguish between three feedback instance categories in segments rated 3 and 4, showing the correct procedure with explicit attention to concepts, substantive evaluation of student answers, and feedback on mathematical practices (e.g., problem-solving, proving.
Conceptual Feedback: Showing the Correct Procedure with Explicit Attention to Concepts. Showing students procedures, especially during individual and group work, was a characteristic of feedback instances labeled "conceptual". That is, they were labeled both "conceptual" and "procedural". The reason was that in addition to being procedural, the feedback addressed the procedure's conceptual underpinnings or linked the procedure to the students' preconceptions. That is, conceptual feedback was most often intertwined with procedural feedback. Also, to a much higher degree than in the segments rated 2, the teachers asked the students follow-up questions while providing feedback and modified their feedback depending on the answers. The next example clearly shows these features.
Example 3: A student is working individually on a task that requires the insertion of x ¼ À 2, y ¼ 1:5, and z ¼ À0:5 into the expression 3xÀ5y z . The student has already done some work has written this in her notebook: À6 À 7:5 À0:5 ¼ 13:5 À0:5 The student puts her hand up to ask the teacher a question.  12 Teacher: No,no,no,try to think of what they are really asking. You have a number that is À 6. That may be, for instance, the temperature. 13 [Teacher proceeds by explaining that À6 À 7:5 models a 7:5 C drop in temperature from one day with À6 C to the next.] 14 Teacher: … and in such situations I have to go left on the number line without messing with the signs. 15 Student: A-ha! 16 Teacher: So, you see, the number line is the best way of understanding calculations with negative numbers. The rules that we have here are only mnemonics to help us after we have understood the values of the numbers. [The teacher further questions the student to ensure that she thinks correctly and answers correctly.] In this example, the teacher chose to show the student how to complete the task while making conceptual connections to the number line and a temperature drop. Similar to the procedural feedback in segments rated 2, the teacher guided the student through a procedure. The utterances in lines 6 and 14 tell the student exactly what she should do to complete the task: go left on the number line from À 6, and divide by À0:5 to obtain a positive answer. The feature that distinguishes this feedback instance from the one in Example 1 is the explicit attention to concepts. The first concept is seen in lines 4 to 6 where the teacher linked the question to a mathematical representation, the number line. The second concept is after line 11 when the student indicated that she was still confused, which prompted the teacher to point to a real-life representation, a drop in temperature. The teacher thus provided two representations that a student may use when thinking about the subtraction of negative numbers.
Not all the teacher's utterances in this example make explicit references to concepts. In line 10, the teacher resorts to a rule to explain the subtraction, multiplication, and division of negative numbers. This illustrates that the feedback instances could contain several of the same features of segments rated 2.
Note that the task was a procedural exercise and that the student asked for a confirmation of a procedural rule. Even though both student and the task focused on rules and procedures, the teacher chose to respond by focusing on conceptual understanding. Thus, the task's nature did not determine the teacher's feedback: The teachers sometimes gave procedural feedback on non-routine tasks and conceptual feedback on procedural tasks.
Conceptual Feedback: Substantive Evaluation of Student Answers. The second most prevalent feature of feedback instances labeled "conceptual" was evaluations which focused on conceptual understanding after the students had finished a task. The teachers' evaluations were focused on why the students' solution methods were correct or incorrect and sometimes on why the solution methods were efficient or elegant. The teachers provided students with evaluative feedback during individual work (Example 4) and whole-class discussions.
Example 4: This example occurred during individual work while the students were solving exercises in the textbook. We understand from the context that the student question in line 1 concerns whether 24 is a common denominator of 8 and 12. 4 Teacher: Ah, but why are you suggesting 24? 5 Student: Because I can make 24 from 8 to 12, like here it is … [inaudible] 6 Teacher Yes, good! You have accomplished the task of making the denominators equal. You do not need to multiply them all the time. Here, you have made them equal, and it is smaller, so that seems to me to be the better option. It will be easier to calculate with later.
In line 6, after listening to what the student was thinking, the teacher commented on why the student's solution was correct. In segments rated 2, the teachers stopped after acknowledging the correct answer (line 2: "Yes, it is."). However, in segments rated 3 and 4, the teachers sometimes continued by asking a question (as in lines 2 and 4) or elaborating on the answer's qualities (line 6).
The teachers sometimes did these evaluations to the whole class. When teachers noticed that many students were struggling to complete a task, they brought the class's attention to a solution used by a student who had completed the task successfully. For the whole class, the teachers spent more time evaluating the student methods and carefully addressing the problems that the class was having.
Feedback on Mathematical Practices. Only one segment showed a teacher consistently giving feedback on mathematical practices: Example 5: The teacher posed a problem to the students: find a mathematical expression showing how many chairs are needed for the table arrangement in Fig. 3, when there are (a) 4 tables, (b) 25 tables, and (c) n tables. The teacher told the students to work in pairs and emphasized that the most important part of the task was to discuss their thoughts with their partner, that the thoughts behind the answers were as important as the answer itself, and that their thoughts must be written down.
While the students worked, the teacher talked to nine pairs of students in turn. First, she asked what they had discussed. After the students answered, she gave suggestions about how to continue, providing feedback according to each pair's specific challenges.
Here are her suggestions to four of the pairs: Teacher: I am wondering if the calculations you are telling me match the expressions you have written in your books. Have you talked to Hedda about this? Go over that again. Together. Check if both agree.
Teacher: You have proposed a formula here. One should check if one's proposals are correct. Continue now by trying to find out if it is correct. How would you argue? Discuss this.
Teacher: Now you have solved (a) and (b) and struggle with (c). It is often a good idea to think about whether the same approach will work or not. […] Discuss what they really mean by n tables.
Teacher: Good that you drew a figure. That's very clever because you can see patterns that are hard to see without [it].
In the four instances, the feedback gives feedback on the students' problem-solving, justification, and mathematical discourse, which are considered important mathematical practices that students should acquire. For example, the feedback in the first instance serves to make the students aware that their answers lack justification and proposes a way forward (i.e., "Discuss this"). The feedback in this segment addressed neither procedures nor concepts, but it evaluated and gave suggestions for the students' engagement in mathematical practices, such as problem-solving, argumentation, and discourse.
There were isolated examples of feedback on mathematical practices in other teachers' lessons. This feedback was often provided to a student who was working on more difficult tasks than those being solved by the rest of the class. For example, in one class, all the students were working on textbook exercises except for one girl who was working on more cognitively demanding tasks that the teacher had assigned her. The teacher provided procedural feedback to all the students except this girl. The teacher provided her the feedback that one answer was not fully justified and, later in the segment, that an answer could be further generalized. Both feedback instances supported the students in the mathematical practices of justification and generalization.
In sum, we have analyzed teachers use of feedback using frequency and distribution of feedback drawing on the Feedback element as defined in PLATO observation manual (step 1), and characteristics of these feedback instances focusing for procedural feedback, conceptual feedback, and feedback on mathematical practices (step 2).

Discussion
The study's first aim was to determine whether the teachers' feedback aligned with generic indicators of quality feedback. In rating every 15 min of instruction (n ¼ 520Þ of 172 lessons using a standardized observation instrument, we found that in about 67% of the segments, the teachers provided only procedural feedback or vague and perfunctory feedback. However, in 19% of the segments, the teachers' feedback included substantive elements. The feedback ratings were unequally distributed among the teachers; 14 of 47 teachers (30%) did not teach a higher-rated segment. Of the 34 teachers (70%) that did teach higher-rated segments, only five (11%) received the highest rating for a segment, meaning that they consistently provided substantive feedback that was likely to help the student.
We warn against overgeneralizing the finding that 30% of teachers never provided feedback other than procedural or vague and perfunctory feedback. Our results do not imply that these teachers did not attend to conceptual understanding or support students' engagement in mathematical practices, only that they did not do so in their feedback. The teachers may have supported these competencies through other instructional practices, such as providing tasks or explanations. Providing procedural feedback may also lessen student's cognitive load, making it easier for them to focus on understanding the concepts. Moreover, the prevalence of procedural feedback during individual work and whole-class instruction should not be assumed to imply that the teaching in these classrooms was "traditional," "teacher centered," or similar. The feedback was often procedural and prescriptive also when the teacher used extended group-work and the tasks were cognitively demanding. Conversely, Example 3 shows that feedback could attend to students' conceptual understanding when the classroom was organized for individual work and whole-class lecturing and the students worked on procedural tasks. To summarize, the teachers' instructional repertoires were varied, but based on the oral feedback analyzed in the present study, some of the classrooms had a narrow procedural focus.
The second aim was to investigate the amount of time teachers spent providing oral feedback in mathematics lessons. By identifying and time-stamping all the feedback instances in a subsample of the 15-min segments (n ¼ 54), we found thatdon averagedthe teachers spent approximately seven and a half out of the 15 min providing feedback. Orally providing feedback to students was thus a frequent teaching practice present across all the teachers and instructional formats. In particular, the teachers constantly provided feedback during individual and group work. When teachers spend so much lesson time providing feedback, it is crucial that students benefit from it. In this regard, it is positive that almost all feedback focused on mathematics and not solely on praise or encouragement, but we can question the usefulness of spending so much time providing feedback that shows students correct solution methods during individual work. Students often had the same problems with the same tasks, and the teachers gave lengthy procedural feedback to each student. The time might be better spent by highlighting and contrasting student solutions in a class discussion (Stein et al., 2008) or by re-teaching the whole class, because "if students lack necessary knowledge, further instruction is more powerful than feedback information" (Hattie & Timperley, 2007, p. 91). The study's third aim was to identify how much and how teachers gave feedback that focused on procedures, concepts, and mathematical practices. The predominant way that the teachers provided feedback across rating levels was to explain procedures during individual work. We expected that the teachers would provide mostly procedural feedback, given that (a) mathematics teaching has a procedural focus in many countries (Hiebert et al., 2003), (b) it is difficult to provide conceptual feedback (Son & Sinclair, 2010), especially "on the spot" in classroom situations (Pankow et al., 2018), and (c) teachers may have been taught mathematics as a collection of procedures themselves so they may not have the required conceptual understanding. However, in 19% of the segments the teachers gave feedback on "substantive elements" that was mainly conceptual feedback. A central feature of how the teachers provided conceptual feedback was that they provided it while showing, explaining, or commenting on a procedure, as in Example 3. Why did the teachers address concepts in their feedback mainly by explaining procedures? Due to the challenges mentioned above, it seems reasonable to speculate that the concrete solution procedures may have served as a scaffold for the teachers in explaining concepts to the students, but further work must be conducted before concluding.
To the best of our knowledge, this study is the first to investigate feedback on mathematical practices across several classrooms (for case studies, see Moschkovich, 2015;Selling, 2016). Although several teachers sometimes provided feedback on mathematical practices to individual students, only one teacher consistently did so (Example 5). It is unsurprising that feedback on mathematical practices is rarely provided because of the lack of research, systematic training, and professional development that would enable mathematics teachers to provide it in the classroom. Now that mathematical practices are a core part of curricula in many countries, research on how to enable teachers to support themdfor example with feedbackdshould follow suit (Schoenfeld, 2015).

Implications for teacher education and professional development
The study has important implications for teacher education and professional development. First, if teacher feedback shall support students in developing the competencies required in newer curricula, teachers must provide feedback on procedural skills, conceptual understanding, and mathematical practices. The results show that helping teachers provide conceptual feedback does not necessarily imply making major changes to their lessons, as procedural feedback and conceptual feedback were provided in similar situations. This does not imply that it will be easy to change teachers' feedback to have a conceptual focus. Teachers themselves often lack strong conceptual knowledge (Hannigan et al., 2013) and this is needed along with pedagogical knowledge to know when a student's struggle stems from conceptual problems and how to create a correct conceptual explanation (Son & Sinclair, 2010).
To further strengthen this practice, teachers and teacher candidates must be trained in how to provide conceptual feedback, not only as a general guideline, but as a specific practice that can be trained, rehearsed, and perfected. Videos and observation instruments are promising tools in this regard. Observation instruments have been used to support teachers in having collaborative conversations about their own practice that lead to lasting changes in teaching (Gore et al., 2017). Reports from using videos in teacher education and professional development that focus on teachers' use of scaffolding techniques such as feedback, suggest that a targeted focus on specific teaching practices strengthens prospective teachers' practical classroom repertoires (Borko et al., 2014;Stockero et al., 2017).
Through the AfL policy initiative, training in feedback has spread to countries around the world (Birenbaum et al., 2015), an example being the nation-wide professional development effort in Norway. Most of these initiatives have not been subject-specific (Bennett, 2011;Birenbaum et al., 2015), and our study suggests that they could benefit from that: both the procedural feedback and the conceptual feedback seen in our study fulfill many of the common prescriptions for high-quality feedback described in seminal reviews like Hattie and Timperley (2007) and Shute (2008). That is, they do not direct attention to the self, they are specific to the students' work, and they contain information about how to improve. However, students may learn very different aspects of mathematics when teachers provide feedback only on the correct execution of procedures and when they also include conceptual links or a focus on mathematical practices. An implication is that teacher trainers can make teachers aware of how classroom feedback tends to focus students' attention on a procedural view of mathematics and provide examples of alternative foci for feedback.

Conclusion
Because of the large number of lessons analyzed, and the focus on the three central aspects of mathematical competency, our findings contribute substantially to the knowledge of classroom feedback practices in mathematics. The study provides knowledge about the prevalence of feedback addressing procedural skills, conceptual understanding, and engagement in mathematical practices and detailed examples to display teachers' feedback repertoires. Such knowledge is useful to ground future research, teacher education and professional development in actual classroom practice.

Funding
This work was funded by the Research Council of Norway. The authors declare that there were no conflicts of interest.