Introduction

One of the most important challenges facing schools is to ensure that students, especially those who are educationally disadvantaged, have the necessary tools and opportunities to engage in higher-order learning goals such as collaborative inquiry. Collaborative inquiry, a major research strand in the field of computer-supported collaborative learning (CSCL), can bring students many benefits such as the development of collaboration skills, agency, critical thinking, metacognition and regulation (Raes et al. 2014; White and Frederiksen 1998; Zohar and Dori 2003). However, many teachers do not believe students with low achievement to be capable of achieving higher-order learning goals (Raes et al. 2014; Zohar et al. 2001). As a result, students with low achievement are likely to be deprived of access to collaborative inquiry, and have few opportunities to benefit from it (Kung and Linder 2007; Schraw 2007). The findings of learning-sciences research suggest that in many cases, students find collaborative inquiry difficult, not primarily because they lack intellectual ability, but because they do not know how to collaborate, how to inquire, how to reflect, or how to learn (Bransford et al. 1999; White and Frederiksen 1998, 2005). Reflective assessment, which incorporates the metacognitive components of planning, monitoring, and reflecting, has been shown effective for scaffolding collaborative inquiry of students with varying levels of assessment (Lee et al. 2006; Toth et al. 2002; White and Frederiksen 1998). Nevertheless, few studies have investigated the performance and understanding of students with low academic achievement in CSCL.

In Hong Kong, students are very competitive and achievement oriented even in primary schools. Secondary schools are classified in three bands—Band 1 (highest) through Band 3 (lowest)—based on achievement on a government examination, of the majority of its Grade 6 students. In this regard, most students in Band-3 schools are students with low achievement, who are not adequately engaged with their schoolwork and academic achievement (Shen et al. 2007). Students with low achievement are often one or more grade levels behind in mathematical skills, language and basic reading (Reglin 1993; Slavin 1991), and are often found to have some learning difficulties (Zohar and Dori 2003) and limited metacognitive skills (Hacker et al. 2000). They exhibit little interest and negative attitudes toward their learning, and a poor self-image. Helping students like these to engage in and benefit from collaborative inquiry is a great challenge for educators. Here, engagement refers to participation, investment or commitment (Marks 2000), or effortful involvement in learning (Pekrun and Linnenbrink-Garcia 2012; Reschly and Christenson 2012).

Recognizing these challenges, this study designed a knowledge-building environment augmented by reflective assessment. In this learning environment, a class of students from a school below the 10th percentile on government examinations, hence labeled “bottom10” by the teachers and students, were facilitated to engage in collaborative knowledge-building discourse aided by reflective assessment. The goal of the study was to explore whether students with low achievement can use and benefit from a reflective-assessment approach to improve their attempts at knowledge building, an influential example of a collaborative inquiry model using CSCL technology. In carrying out reflective assessment, students used an assessment tool, the Knowledge Connections Analyzer (KCA; van Aalst et al. 2012), which collects information pertaining to several intuitive questions about online discourse from the Knowledge Forum® database.

Background

Knowledge building for students with low achievement

Many detailed accounts of knowledge building, including its underlying principles, are available in the learning-sciences literature (Scardamalia and Bereiter 2014; Scardmalia 2002; van Aalst 2009; Zhang et al. 2007). Here we describe only the features of knowledge building most pertinent to the study. In knowledge building, a class of students works together as a community to advance the state of knowledge in that community, as they perceive it. Although the students do not necessarily make groundbreaking advances to human knowledge, they do surpass what they knew together when they began and what is described in the sources they study (Bereiter and Scardamalia 2010). Among the most important principles of knowledge building are collective responsibility for community knowledge, epistemic agency, constructive use of authoritative sources, idea improvement, and synthesis and “rise-above.” These are primarily accomplished through progressive discourse, typically online discourse on Knowledge Forum®. Another important aspect of knowledge building is the principle embedded and transformative assessment (Scardmalia 2002), which posits that assessment is part of the knowledge-building process and transforms it. This study examines this kind of assessment but primarily targets collective responsibility for community knowledge, for which we use the proxy “Are we a community that collaborates?” and synthesis/rise-above, for which we use the proxy “Are we putting our ideas together?” (van Aalst et al. 2012).

In the last two decades, knowledge building has been shown to enhance the performance and learning of learners of various ages (Chuy et al. 2010; Niu and van Aalst 2009; So et al. 2010; van Aalst and Truong 2011; Zhang et al. 2007; Authors 2014). However, can knowledge building benefit students with low achievement? Successfully engaging students at various levels in the pursuit of higher-order learning goals is an important area of learning-sciences research. However, few researchers have focused on students with low achievement. White and Frederiksen (1998) found that both high-achieving students and students with low achievement benefited from collaborative inquiry in which students engaged in continuous assessment of and reflection on their performance and inquiry process, but observed a larger net achievement gain among students with low achievement. Zohar and Dori (2003) provided additional evidence of the generally positive influence of collaborative inquiry on the performance of high-achieving students and students with low achievement. In one study, they found that students with low achievement gained significantly more from reflective inquiry than their high-achieving counterparts. Similarly, Raes et al. (2014) found Web-based collaborative inquiry to benefit both high achievers and students with low achievement, but showed that significantly greater gains were made by students with low achievement.

We found no published studies of knowledge building that specifically investigate its effects on students with low achievement. We identified only three studies that investigated its effects on both students with high achievement and low achievement in primary and secondary schools. Chan and Lee (2007) examined the effects of embedded portfolio assessment on the performance of high-average students at a Hong Kong high school. They found that portfolio assessment guided by knowledge-building principles had a more positive influence on students’ conceptual understanding than portfolio assessment without a foundation in knowledge-building principles, and that this influence was stronger for students with lower achievement. Niu and van Aalst (2009) measured the benefits of knowledge building for both honors students and students in regular classes at a Canadian high school. The students in the honors class outperformed those in the regular class, but the within-class differences were larger than the between-class differences. In addition, the students in the regular class outperformed those in the honors class on some of the qualitative measures. Finally, So et al. (2010) examined the effects of the knowledge-building model on the performance of students at various levels of achievement at a Singaporean primary school. Most of the students came from non-English-speaking homes with a low to middle socio-economic status. The researchers found that knowledge building benefited both high achievers and students with low achievement, but that the students experienced difficulties in developing metacognitive skills such as monitoring and reflection, which are critical to productive knowledge building. The researchers called for further studies to identify strategies and methods of scaffolding students’ development and deployment of metacognitive skills. In summary, the findings of the above studies suggest that both high-achieving students and students with low achievement can benefit from a supportive knowledge-building environment. Nevertheless, few published studies have investigated the performance of students with very low academic achievement—e.g., students who score below the 33th percentile on national or territory-wide public examinations.

Further, engaging students in sustained inquiry remains challenging. One difficulty is the relative lack of synthesis and rise-above of ideas, which are critical to productive knowledge building. With student ideas distributed across individual postings over time, it is difficult for students to understand the conceptual landscape, causing short discussions that lack conceptual progress and knowledge integration (Suthers et al. 2008; Zhang 2009). Therefore, appropriate tools and designs, and particularly scaffoldings should be provided to help students to engage in ongoing review of and reflection on collective advances and gaps, and to regularly summarize and synthesize ideas.

Metacognition

Metacognition was originally defined by Flavell (1979) as “thinking about thinking”. Later, Brown (1987) identified two types of expertise: knowledge about cognition, and self-regulation for managing and improving cognition. Knowledge about cognition includes knowing and reflecting on what one knows and does not know, and identifying such gaps. Self-regulation refers to students’ use of strategies that include planning, monitoring, and reflection to manage and improve cognition (White et al. 2009). Planning involves selection of appropriate strategies and allocation of resources to organize and prepare for an upcoming task (Didonato 2013). Research examining skilled writers found that better writers spent more time planning and setting goals prior to carrying out the task, that and they were better able to achieve their plans compared with poorer writers (Bereiter and Scardamalia 1987). Monitoring refers to assessing and evaluating a range of things including inquiry process and products, progress at achieving goals, and metacognition itself. Monitoring can be achieved by getting students to ask themselves questions (White et al. 2009), and to compare their performance with their criteria or learning goals. Reflection is defined as “intellectual and affective activities in which individuals engage to explore their experiences in order to lead to new understandings and appreciations” (Boud et al. 1985, p. 19). Reflection can provide opportunities for improving the inquiry process (White et al. 2009).

Metacognition plays a crucial role in the development of the various capabilities required for higher-order learning goals (White and Frederiksen 2005; Zohar and Dori 2003), as well as the transformation of students and classrooms into self-aware and self-improving systems. “[Students] will create theories about what they are doing and why as they constantly engage in cycles of planning, monitoring, reflecting and improving” (White and Frederiksen 2005, p. 211). Previous research on collaborative inquiry has shown that productive learning depends in part on students’ development, deployment, and adaptation of metacognitive skills (Järvelä and Hadwin 2013; Schwarz and White 2005). There is also evidence that the use of metacognitive skills can help students to improve collaborative inquiry processes and domain understanding (Chan et al. 2012; van Aalst and Chan 2007); and leads to high-quality learning and transfer (Didonato 2013; Vauras et al. 2003). With appropriate scaffolding, students do engage in metacognitive processes as they work collaboratively on problems (Hurme and Järvelä 2005).

Many children, and some adults, however, often lack metacognitive skills, which develop slowly (Pressley and Ghatala 1990) and are observed less often among students with low achievement than their high-achieving counterparts (Hacker et al. 2000). For instance, De Jong et al. (2005) found that middle-school students working in a CSCL environment helped each other to engage in learning not primarily by planning or monitoring, but by maintaining common ground and using cognitive strategies. A similar pattern was reported by Volet et al. (2009). Finally, in a collaborative learning study that engaged students with low achievement in metacognitive learning, Azevedo et al. (2004) found that students with low achievement made statistically significant gains from pretest to posttest, but were likely to use low-level strategies, such as searching the environment and following procedural tasks, rather than planning or monitoring their activities. Therefore, although these studies highlight the potential of metacognition to improve learning and performance outcomes, they also show that high-quality metacognition is not easily achieved.

Furthermore, metacognition and regulation is distributed among individuals in collaborative inquiry (Järvelä et al. 2015). Students promote partners to engage in metacognitive processing by asking questions and requesting explanations (Hakkarainen 2003; Hmelo-Silver and Barrows 2008; Lee et al. 2006; Poitras et al. 2012), and evaluating and reflecting on group progress (Chan 2012). Performing metacognitive activities individually is difficult for students; typically, students with low achievement do not monitor how well they have achieved the task goals, or question whether or not they understand. However, in collaborative problem-solving contexts, with distributed metacognitive skills, students could facilitate partners to carry out metacognitive activities (Chan 2012).

Reflective assessment

Reflective assessment refers to assessment in which students are asked to reflect on a set of criteria/principles or learning goals, to generate their own feedback based on their continuous assessment of and reflection on the inquiry process and product, and to improve their ongoing learning in the form of grappling with broader problems and continuously creating knowledge (Scardmalia 2002; White and Frederiksen 1998). Reflective assessment is similar to embedded and transformative assessment proposed by Scardmalia (2002): “Assessment is part of the effort to advance knowledge—it is used to identify problems as the work proceeds and is embedded in the day-to-day workings of the [class]” (p. 82). Reflective assessment is an ongoing and integral component of learning: it does not merely evaluate what students have done and point to the next predetermined step, but involves student directing their efforts to improve their ongoing learning. With reflective assessment, students deploy and develop metacognitive reflection incorporating the metacognitive components of thinking about what they know and need to know, and regulation strategies of planning, monitoring, and reflecting on their understanding. Further, reflective assessment is collaborative: not everyone in the community needs to be highly metacognitive for the process to be effective, and students can scaffold one another’s metacognitive development through social modeling. Reflective assessment is similar to, yet different from, formative assessment, which is performed during the learning process in order to enhance the learning process (Black and Wiliam 1998). However, formative assessment is usually controlled by the teacher and oriented toward a predetermined endpoint (Taras 2009); students usually respond to formative feedback (Ramaprasad 1993) from their teacher rather than play an active role in generating the feedback (van Aalst 2013).

Prior research on knowledge building and reflective assessment suggests that students of various achievement levels can participate in and benefit from reflective assessment (Herrenkohl et al. 2011; Lee et al. 2006; Schwarz and White 2005; Toth et al. 2002; van Aalst and Chan 2007; White and Frederiksen 1998). For instance, previous research by the authors (Lee et al. 2006; van Aalst and Chan 2007) on the use of electronic portfolios in Knowledge Forum, in which students self-assessed and reflected on their performance in knowledge-building, has revealed the scaffolding function of this type of assessment. Through carrying out this task, students gained a better understanding of both the nature of knowledge building and how they should work on Knowledge Forum (van Aalst and Chan 2007). However, although these studies involved students at multiple grade levels (Grade 9 and Grade 12), most of the students had average or higher than average levels of prior achievement. Nevertheless, studies on reflective assessment that have used other methods suggest that below-average students can benefit from structured but reflective assessment, particularly the well-known study of White and Frederiksen (1998), which showed that such students benefitted more from reflective assessment than above-average students.

This study aimed to help students with low achievement to engage in sustained knowledge building through reflective assessment; it was part of a large project that involves a series of case studies to investigate the design, process and effects of reflective assessment on the improvement of students’ attempt at knowledge building, using the Knowledge Connection Analyzer (KCA) an assessment tool developed by our reseach team (van Aalst et al. 2012).

Reflective assessment is epistemologically similar to other, domain-specific inquiries; it is not merely reflection based on opinions, but data-informed discourse and problem solving. Rendering it feasible in a collaborative online environment, however, requires tools that collect data automatically and present them in a ready-for-use format (Strijbos 2011). This study investigated whether and how students with low achievement could engage in reflective assessment in knowledge building, supported by the Knowledge Connection Analyzer (KCA).

The Knowledge Connections Analyzer (KCA)

The Knowledge Connection Analyzer (KCA) is a Web-based assessment tool designed to be used by students to reflect on their online work in Knowledge Forum. It queries the Knowledge Forum database to collect information on four intuitive questions related to knowledge building. Table 1 shows the questions, their purposes, and the corresponding knowledge-building principles and perspectives. In most cases, the KCA displays quantitative data in pie charts and bar graphs and qualitative information in text-based tables. For a full description, see van Aalst et al. (2012).Footnote 1 In this study, we focused on the first two questions, which draw attention to the social and collective nature of knowledge building; it was not possible, within the available instructional time, for the students to use all four questions. Nevertheless, the two questions studied here are very important because assessment practices are generally dominated by attention to the performance of individual students, and not collective performance.

Table 1 Embedded knowledge-building principles in the four questions
Full size table

The first question (“Are we a community that collaborates?”) intended to prompt the students and teachers to focus on the idea that knowledge building has a community-level goal. The KCA analyzes this question using an analogy to social media concepts such as friends and followers. If a class is operating as a community, we would expect most of the students in that class to have networks of other students who read, respond to, or use some of their notes. Therefore, it is necessary to specify the size of the network and the number of interactions that occur between the target students and every other student in the network that are of interest. For example, the following question could be asked: “How many students have received at least one build-on note (response) from five students?” The KCA shows the results in a pie chart and provides information on all of the students who meet the specified criterion.

The second question (“Are we putting our knowledge together?”) also focuses on the community, but on the extent to which students are linking ideas and synthesizing what they are learning about different problems. In Knowledge Forum, these kinds of activity are evident from the number of hyperlinks between notes and the number of rise-above notes. As the Knowledge Connection Analyzer (KCA) displays data both textually and in pie-chart format, students can select a note with many links and check the content of the linked notes to evaluate the appropriateness of the links.

In this study, we explored the potential of reflective assessment to help students with low achievement to collectively improve their knowledge-building discourse. This study focused on the social/collective aspects of knowledge building, which are notoriously difficult to bring into focus in assessment; almost all previous scholars have examined the performance of individual students. This study is an extension of previous research on portfolio assessment, one type of reflective assessment, of knowledge building (Lee et al. 2006; van Aalst and Chan 2007; Zhao and Chan 2014). The following research questions were investigated:

  1. 1.

    Did the collaboration and synthesis change during the knowledge-building process?

  2. 2.

    What was the nature of knowledge-building discourse? To what extent did students with low achievement improve their knowledge-building discourse?

  3. 3.

    How did students with low achievement carry out reflective assessment to reflect on and improve their ongoing knowledge-building discourse by using the Knowledge Connection Analyzer (KCA)?

Methods

Study design

We investigated the discourse in Knowledge Forum of a “bottom 10” Hong Kong class and the critical events related to the use of reflective assessment. First, this study used the Knowledge Connection Analyzer (KCA) to analyze the development of the database, that is, using the same information as the students. Next, we analyzed the Knowledge Forum database using methods that have been employed in many previous studies of knowledge building (van Aalst and Truong 2011; Zhang et al. 2007). We identified inquiry threads, which are networks of notes on distinct problems or topics (Zhang et al. 2007), and qualitatively analyzed the discourse in these threads. In this we made use of the coding scheme of van Aalst (2009) distinguishing between knowledge sharing, knowledge construction, and knowledge building/creation, and a coding framework we developed for tracing student interactions within the inquiry threads. Finally, we identified the critical events of reflective assessment that may have contributed to students based on analysis of a set of qualitative data such as classroom observations, video-recorded reflective-assessment sessions, student interviews, and students’ artifacts.

Research context and participants

The study was conducted at a Band-3 school in Hong Kong. The school was not only a Band-3 school, but students at the school scored below the 10thpercentile on a territory-wide examination. The participants comprised 20 students in an 11th-grade class taking a visual-arts course; they were taught in Chinese, as is typical in schools with this level of performance. The teacher with more than 25 years of experience had previously taught in higher-banding schools but chose to work in a band-three bottom-ten school with his strong belief that he can do more innovative work with weak students. The teacher found the students to be more motivated by knowledge building, a promising student-centered approach, than by other teaching approaches. With the public examination requirements, he needed to develop ways to have students engage, inquire, and create ideas, thus a community approach was appropriate. The students had used Knowledge Forum for 2 months prior to the study, mainly to prepare for their formal school-based assessments, which are combined with the government examination results. The teacher had considerable experience teaching visual arts and had taken a postgraduate course on knowledge building; he had used knowledge building for approximately 6 years and engaged in considerable professional teacher development in a teacher network on knowledge building (Chan 2011). While the teacher was clearly exceptional in the above respects, the approach he used to design the knowledge-building environment has been used by many other teachers in Hong Kong.

Curriculum and pedagogical design

Course design

The main objective of the course was to help the students to develop creative and inquiry skills in relation to the visual arts. The topic of inquiry was “Design,” one of the core components of the visual-arts curriculum; it lasted approximately 4–5 months (January–June; one lesson per week). The work comprised whole-class discussion in the classroom, small-group discussion, individual and collaborative note writing, and similar activities. To incorporate online discourse into the regular course, the teacher occasionally provided the students with articles from magazines and the Internet, and frequently organized class discussions to enable the students to engage with key questions addressed in the course syllabus, such as the following: “What is design, and how do designs differ from imagined things?” “How do we evaluate design?” “What are the relationships between design, history, and technology?” “What are the effects of design?” The organization of the course materials was shaped by the students’ emergent inquiry. While working on Knowledge Forum in class, some of the students either individually or collectively used the Knowledge Connection Analyzer (KCA) and corresponding prompt sheets to carry out reflective assessment. Different students were responsible for performing the reflective assessment in each lesson. We designed prompt sheets for KCA-reflection tasks collaboratively with the teacher. As typical in knowledge-building research, the teacher and researchers worked together as co-investigators in the curriculum and pedagogical design.

Pedagogical design

The study was pedagogically designed to accommodate diversity in academic ability and support knowledge building to help students to develop creative and inquiry skills. The teacher used a pedagogical process consisting of three key components to familiarize the students with knowledge building, as described in detail by Chan (2011), and van Aalst and Chan (2012). These scaffolding include idea-focused pedagogy emphasized in knowledge building, but also adapted to help accommodate to low achievers. Briefly, the three components were implemented as follows.

Helping students to develop inquiry and collaborative capabilities (weeks 1–5)

To increase the students’ motivation and enhance their communication and collaboration skills, small groups of students were asked to collaboratively construct three-dimensional mind-maps from wires of various colors. Each mind-map represented the students’ ideas about a particular topic of inquiry. They then worked as a community in constructing a “knowledge wall” before working on Knowledge Forum, augmented by whole-class and small-group discussion. A knowledge wall is a visual representation of ideas attached physically to a wall; index cards represent ideas and string is used to express connections between them. Visually, the effect is similar to that of a discussion space (“view”) in Knowledge Forum. The teacher guided the students to develop high-quality questions and explanations through Socratic and peer dialogue.

Developing problem-centered inquiry on Knowledge Forum (weeks 6–8)

The students deepened their inquiry and improved the ideas displayed on the knowledge wall by designing authentic questions in small groups with the help of a rubric. Next, the questions were presented, and the most interesting were selected by the students and input into Knowledge Forum for further inquiry. The teacher guided knowledge building by suggesting gaps for further inquiry, integrating classroom discussion with the students’ work on Knowledge Forum.

Deepening deep domain understanding and knowledge advances through reflective assessment (weeks 9–17)

After working on the Knowledge Forum database for approximately 3 weeks, the students were guided to write high-quality notes by making reference to knowledge-building principles, and were prompted to contribute more notes through reflection on the existing assessment tools in Knowledge Forum. The students then created both group and individual portfolio notes, and used the KCA data to reflect on their online discourse individually and collectively. The prompts were both content-related and metacognitive, and corresponded to each of the four questions in the KCA.

Data sources

Classroom observations

We conducted the observations while the participants were engaged in inquiries related to knowledge building. These collaborative inquiry activities included small-group work, whole-class conversations and discussion, knowledge building talk, reflective assessment, and computer-based sessions with Knowledge Forum and the Knowledge Connection Analyzer (KCA). The data comprised photographs of and field notes on some of the lessons, and video recordings of approximately 10 1-h lessons.

Artifacts of students’ work

These comprised the Knowledge Forum database, the students’ prompt sheets, and the results of the students’ the KCA analysis. In 2.5 months, the students wrote 161 computer notes, including 13 portfolio notes. The students’ prompt sheets that we designed comprised reflective journal prompt sheets (Appendix A) and the KCA prompt sheets (Appendix B). The latter were used to record the students’ multiple sets of KCA results, their interpretations of and reflections on the data, and their action planning. To enhance the students’ metacognitive activities and thinking, we provided students a reflective journal prompt that included a set of metacognitive prompts, such as “My Analysis,” “My Motivation,” “My Problem,” and “My Plan.” We also provided students the KCA prompt sheets that provided a set of content-related and metacognitive prompts for each of the four questions in the KCA. The prompt sheets were designed to help the students to make sense of the quantitative data generated by the KCA, and to encourage them to engage in metacognitive activities (e.g., monitoring, reflection, and regulation). Together, these prompts formed a metacognitive model that fostered the students’ engagement in a series of metacognitive activities, such as conducting self-or peer assessment, writing reflections, establishing learning goals, and continuously improving their work. The KCA prompt sheets were distributed to the participating students in each lesson, and collected a week later. Some of the students finished their reflection journals and prompt sheets in class, while others completed these tasks after class.

Interviews

Semi-structured interviews were used to obtain information on the students’ experiences of using the Knowledge Connection Analyzer (KCA). Some of the interviews were conducted individually, and others in groups of 2–3 students. Most of the interviews were conducted informally, in the computer laboratory in which the students used the KCA during class, and each lasted for approximately 20–30 min. The interview questions corresponded to the four questions in the KCA to obtain information on students’ use of the KCA to support reflective assessment. The interviews were audio recorded. The data obtained in the interviews helped us to understand both the students’ use of assessment data before undertaking the KCA and their KCA-related activities.

Questionnaire

To probe the students’ perceptions of their KCA-related experiences, a short questionnaire was administered after the first use of the KCA. The students were asked to share their ideas about the benefits and challenges of the KCA. Among the questions asked were: “Are the results generated by the KCA easy to understand? If not, which aspects are difficult to understand?” “How did you reflect on your own notes before using the KCA?” “Will you use the KCA results to reflect on your knowledge-building notes?” “Does the KCA help you to write notes?” “Do you have any suggestions for the developers of the KCA?”

Data analysis

Identifying inquiry threads

All computer notes except the portfolio notes (148 notes) were put into inquiry threads after thematic analysis. An inquiry thread is a conceptual thread of notes that aims to address the same principal problem (Zhang et al. 2007). Inquiry thread analysis is a way of reconstructing the original conversation threads to understand what students, as a community, are trying to achieve and to identify the inquiry topics that students are discussing. To check the coding reliability of the inquiry thread analysis, two raters independently carried out the same process of analysis with 40 % of the notes. They independently identified the principal problems (e.g., intellectual curiosity, characteristics of a good design), and clustered the notes under these principal problems. The inter-rater reliability was .80 (Cohen’s kappa). We further classified the discourse in the inquiry threads using a coding scheme developed by van Aalst (2009), which distinguishes between three types of discourse: knowledge sharing, knowledge construction, and knowledge building/creation. Knowledge sharing involves the accumulation of ideas without reaching agreement or solving problems. Knowledge construction is the development of explanations by means of problem solving, and involves constructive use of information, questions, and explanations, and the co-construction of ideas in problem solving. Knowledge creation/building threads reflect emergent inquiry questions, emerging knowledge creation with meta-discourse, and awareness of community dynamics with reference notes. In the knowledge-building threads, the ideas are substantially improved. Two coders independently classified all of the inquiry threads, obtaining an inter-rater reliability of .77 (Cohen’s kappa).

Analyzing interactions within inquiry threads

To characterize the students’ interactions within and contributions to the discourse at a more granular level, we coded students’ notes in each inquiry thread by using a coding framework (Table 2), with individual notes as the units of analysis. The development of the coding scheme was an iterative process driven by both theory and data. A preliminary set of categories was refined interactively until a set of empirically derived categories was identified. The coding scheme was based on theories of the social dynamics of knowledge building (Fu 2014; van Aalst 2009), the socio-cognitive dynamics of knowledge building (Zhang et al. 2007), and the social, cognitive, and meta-cognitive processes of knowledge construction (Hmelo-Silver and Barrows 2008; Hurme et al. 2006). The three sets of categories were underpinned by socio-cognitive dynamics adopted from knowledge-building principles such as epistemic agency, community knowledge, improvable ideas, and embedded and transformative assessment. The data analysis was iterative: the coders moved back and forth between the codes and the data until saturation was achieved. To gauge the reliability of the qualitative analysis, two raters independently coded the notes (n = 51, 30 %) from two inquiry threads. The inter-rater reliability was .78 for questions, .78 for ideas, and .79 for community (Cohen’s kappas).

Table 2 Coding framework for content analysis of discourse in each inquiry thread
Full size table

Analyzing and identifying events of reflective assessment

To understand the process dynamics of reflective assessment that may have contributed to students productive, the set of qualitative data including classroom observations, video-recordings of reflective-assessment sessions, the Knowledge Connection Analyzer (KCA) prompt sheets, student interviews, and questionnaires was analyzed. We identified many important events, however, we only selected a limited number of events based on the key knowledge-building goals (e.g., community knowledge, improvable ideas, synthesis/rise-above) and the questions on the KCA that fostered reflective assessment.

The multifaceted analysis results of students’ online discourse provided the basis for the qualitative analysis of the set of data. We began analysis by examining the students’ reflective journals and KCA prompt sheets to identify productive and unproductive uses of the KCA. Next, we analyzed the potential of reflective assessment to increase students’ focus on the key goals of knowledge building. We further systematically analyzed the remaining sources of data to cast light on the students’ understanding of the effectiveness of the KCA in supporting reflective assessment, their experiences of using the KCA to conduct reflective assessment.

The trustworthiness was enhanced through consistent observation, methodological triangulation, and a rigorous coding process. We observed the participants consistently for approximately 4-5 months. At the same time, we obtained necessary information about the students, the teacher, methods of instruction, and curriculum by observing the teacher’s work in other classes, and drawing on our 6-year research relationship with the teacher. We conducted methodological triangulation using multiple sources of data: observation, students’ artifacts, interviews, and questionnaires. We attempted to engage in a rigorous coding process by presenting the preliminary findings to our whole research team, and by using their comments and feedback to improve the next round of data analysis. Feedback was solicited from the research team in three separate meetings.

Results

Research question 1: Did the collaboration and synthesis change during the knowledge-building process?

In this section, we use the data provided by the Knowledge Connection Analyzer (KCA) to present a quantitative overview of the development of the students’ collaboration and synthesis before presenting the results of the qualitative analysis of the students’ online discourse.

For reading, we set the KCA to calculate the percentage of students with at least 5 readers who had read at least three of the student’s notes. Having five readers is not very different from the number of listeners a student would typically have during small-group collaboration (perhaps double). However, students usually work together for the duration of a small-group collaboration task, whereas groups of readers on Knowledge Forum are emergent. As building on notes is less common than reading, we set the KCA to calculate the percentage of students who had received at least one build-on note from each of three or more community members. Table 3 shows the results of the measurements, which were taken at approximately 10-day intervals. The results were presented in pie charts for each interval by the KCA, but are aggregated here to facilitate comparison.

Table 3 Changes in student collaboration and synthesis
Full size table

The results show a pattern of increase in all of the indicators. For example, 35 % of students had 5 readers after 9 days of working on the database, and from May 20 onward, 90 % of the students had 5 readers. For being a reader, the percentage increased much more quickly, reaching 80 % after the first 9 days; this indicator also shows that the vast majority of the students were in reading networks comprising at least 5 students.

Table 3 also shows that the class initially created few reference links in their notes; before May 10, fewer than 5 % of the notes included references. The percentage of notes containing reference links increased dramatically after May 10, exceeding 20 % by the end of the study. It is clear from the final column of the table that more than 40 % of the notes were eventually used as references in other notes; some links were created to the 13 portfolio notes. However, the large number of notes used as references suggests that the students remained fairly undiscriminating in their choice of notes to which to link in their portfolio notes.

In summary, the quantitative indicators of the students’ collaboration indicate that the classroom community became more interactive over time. The quantitative indicators of the students’ synthesis suggest that the students began to make connections between notes that were not direct responses.

Research question 2: What was the nature of knowledge-building discourse? To what extent did students with low achievement improve their knowledge-building discourse?

We first present the results of the inquiry thread analysis, followed by the results of the qualitative analysis of the students’ interactions within and contributions to the discourse in each inquiry thread. Then we report the results of characteristics of students’ notes before (weeks 6–8, Time 1) and during (weeks 9–17, Time 2) reflective assessment using the KCA, and examine the differences between the two phases to evaluate the advancement of the knowledge-building discourse.

Inquiry thread analysis

All 148 notes were classified, yielding 9 inquiry threads, as shown in Fig. 1. The number of notes and the number of authors for each thread are shown in parentheses. In some of the threads (#1, #3, #5, #6, #7, and #9), most students were authors, whereas others involved only a small number of authors; this suggests that some problems attracted more attention from the community than others. Most inquiry threads lasted longer than 7 weeks, which suggests that a number of students remained interested in these topics for some time. The dotted lines in Fig. 1 indicate notes that belong to more than one inquiry thread; these “bridging notes” (Zhang et al. 2007) reflect the integration and diffusion of ideas between and across threads.

Fig. 1
figure 1

Network of inquiry threads in knowledge forum. The number following the code indicates the number of notes and the number of authors, respectively

Full size image

Progressive problem solving was evident in many of the nine inquiry threads (e.g., #2, #3, #5, and #9). In these threads, the students proposed theories and problems of interest, regularly monitored and regulated their inquiry by asking relevant questions and seeking clarification and explanation, addressed problems at increasing levels of complexity, and formulated higher levels of conceptualization. In the thread on good design (thread #2), for example, the students first asked whether designs serve particular purposes, which led ultimately to the understanding that the functions of a design are various, flexible, and influenced by many factors. Building on this foundation, the students generated further problems and statements highlighting and addressing the gaps in their understanding, such as the following: “What are the characteristics of a good design?” (S.18) “What do you mean?... Those characteristics are not all related to the three dimensions [of good design] that you proposed.” (S.19) “Good designs should be serviceable and beautiful, and help people to solve problems.” (S.18).

To further examine the degree of knowledge advancement within an inquiry thread, we placed the inquiry threads into three categories: knowledge sharing, knowledge construction, or knowledge building (van Aalst 2009). One of the nine inquiry threads was classified as a knowledge-sharing thread, four as knowledge-construction threads, and four as knowledge-building threads. This is a relatively positive result compared to previous studies (Siqin et al. 2015; van Aalst 2009), and suggests that the students progressively solved problems and improved their ideas in the communal space.

Qualitative analysis of students’ interactions within and contributions to inquiry threads

To characterize how students interacted with each other to identify and address problems, regulate their discourse, and contribute to their community, we qualitatively analyzed the students’ notes in each inquiry thread at a more granular level, using the coding scheme presented in Table 2.

The results of coding the discourses in the inquiry threads are shown in Tables 4 and 5. Generally, the results displayed in Table 4 are consistent with the overall classification of the inquiry threads, and suggest that the students were involved in explanation-oriented discourse. For example, the students created more explanatory than factual questions (18, compared with 11), and wrote more explanatory notes than notes containing simple statements (48 notes, compared with 14). Questions and statements that were more explanatory appeared most frequently in the threads that concerned explanatory issues. In thread #5, for example, the students discussed ways of differentiating designs from imagined things. In the knowledge-building process, much more than problem solving and ideation were involved in discourse. The students needed to regularly reflect on their progress, make connections between new and prior knowledge, formulate deeper levels of conceptualization (known as “rise-above”), and propose new long-range goals for further inquiry. The students asked many metacognitive questions, and also contributed a reasonable number of metacognitive statements, which included meta-discourse (14 notes, not including the portfolio notes) to reflect on their progress and highlighted promising ideas or problems for further inquiry. All of these data indicate that the students invested much effort to assess and reflect on their online discourse. However, the proportions of metacognitive questions and metacognitive statements differed between the inquiry threads, depending on the students’ engagement with the discourse and the nature of the issues they addressed. Metacognitive questions and metacognitive statements appeared more frequently in the threads that concerned explanatory issues.

Table 4 Number of problems and ideas in inquiry threads
Full size table
Table 5 Number of different categories of ideas and community in inquiry threads
Full size table

As shown in Table 5, 72 notes were classified as negotiating a fit, and 9 notes were rise-above notes that extended the referenced ideas and introduced a new level of conceptualization. These results indicate a high frequency of responses to questions and to each other’s ideas in the communal knowledge space; most of the responses negotiated a fit to advance knowledge and created a knowledge space of value to the community and individual students. The results suggest that the students appeared to have made collaborative efforts and engaged in collaborative knowledge building. However, with the exception of portfolio notes, they created few rise-above notes to rise above the ideas in the communal space.

Overall, the above results suggest that in a supportive knowledge-building environment, students in this class appeared to be able to assume the high-level responsibility for collectively accomplishing knowledge-building discourse. The students involved in this study appeared to have engaged in productive interactions and progressively advanced ideas in the communal space.

Questioning, ideation, metacognition, and community

To investigate the advancement of the knowledge-building discourse, we qualitatively analyzed the characteristics of the notes before (weeks 6–8; see the section on pedagogical design; Time 1) and during reflective assessment using the KCA (weeks 9–17; Time 2), based on the qualitative analysis of the students’ interactions within and contributions to the inquiry threads. The proportion of each category of notes was calculated, followed by a Chi-square test, performed to examine possible differences between the two phases.

The students contributed 102 notes in Time 1 and 59 notes in Time 2. The results are shown in Table 6. The frequency distributions were significantly different for the two phases, χ2(df = 8, N = 161) = 46.2, ϕ = .54; this is a moderate to large effect. The results indicate primarily that the students contributed relatively more explanation-seeking questions, explanations, and metacognitive questions and statements in Time 2 than in Time 1, and that the students were mostly engaged in negotiating fits and synthesizing their notes during Time 2.

Table 6 Frequency and percentages of notes classified as having questions, ideas, metacognition and community during early and later phases
Full size table

Research question 3: How did students with low achievement carry out reflective assessment to reflect on and improve their ongoing knowledge-building discourse by using the Knowledge Connection Analyzer (KCA)?

The goal of this analysis was to investigate whether students were able to and how they carried out reflective assessment in knowledge-building process, using the KCA.

The Knowledge Connection Analyzer (KCA) was introduced to help the students to assess, reflect on, and collaboratively improve their knowledge-building discourse. However, many of the students were initially unable to interpret the KCA data productively. The results of a survey conducted with the first nine users of the KCA immediately after its introduction indicated that many of the students noticed only the quantitative data generated by the KCA; they did not recognize the cognitive opportunities afforded by the qualitative information provided. However, the majority (seven) of the nine students indicated their interest in using the KCA and recognized the usefulness of the tool. They reported using the KCA data to reflect on their individual performance and their interactions with other students. They recognized the potential of these data to help them identify their own strengths and weaknesses, as well as areas for future improvement.

After working on Knowledge Forum for three weeks, the students used the Knowledge Connection Analyzer (KCA) to carry out a series of individual and collaborative inquiry supported by the KCA prompt sheets. In addition to the prompt sheets, the students received some support from the research team, such as explanations of the meaning of the KCA output. Qualitative analysis results indicate that reflective assessment using the KCA might have enabled students to focus on the main learning goals of knowledge building such as idea improvement, synthesis and rise-above, community knowledge, and further to improve their knowledge-building discourse. We present the key events to illustrate how students performed reflective assessment in knowledge building using the KCA.

Fostering community orientation through reflective assessment using the KCA

The question “Are we a community that collaborates?” was intended to help students think about knowledge building as something a community does together rather than individually. The following interview excerptFootnote 2 shows how one student reflected on the data from this question:

...I can see my deficiencies...For example, reading others’ notes, I have read many notes, the percentage is 99% which is pretty high. However, when I look at built-on to others’ notes, the percentage is only about 20%, which is pretty low... The data tells me that the interaction between the other students and me is not sufficient. I need to communicate more with peers... I even write notes at home, because I am afraid that my point or idea is not good enough...(S.15, from an interview immediately after the student analyzing the data from the question “Are we a community that collaborates?” in the KCA)

From the above excerpt, we found that this student analyzed the Knowledge Connection Analyzer (KCA) data and identified the gaps (“have read many notes;” “the percentage of my response to others is only about 20%, which is pretty low;. ..built-on... the percentage is only about 20%, which is pretty low;” “the interaction between the other students and me is not sufficient”). This demonstrates useful insight as the student can now see that he has read more but built-on less; the students also thought that the interaction alone was insufficient. Based on his analysis, the student appeared to reflect on the quality and importantly generate further plans to bridge the gaps (“communicate more with peers;”“...I am afraid that my point or idea is not good enough”). The student not only thought about the number—he will not “cheat” the system by mindlessly building on notes—but generated useful knowledge building ideas including improving ideas. However, this example also illustrates that the student reflected on the KCA data from his individual rather than the community’s perspective: focusing on his own performance and how the others interacted with him, and generating individual plans aimed to close the identified gaps. This suggests that scaffolding is necessary to bring the community orientation intended with “Are we a community that collaborates?” into focus for students.

Promoting synthesis and rise-above as collective responsibility through reflective assessment using the KCA

The students were scaffolded to synthesize and rise above ideas collectively through reflecting on data from the question “Are we putting our knowledge together?” The following is an example of an excerpt from the Knowledge Connection Analyzer (KCA) task sheet of one student:

I am wondering whether we have really read the notes carefully and applied them to date. The number of reference notes is 34 and the number of notes that are being used as references is 56. We have written more than 150 notes...These data tell us that the extent of synthesis and collaboration is not higher than our expectation... (S.18, from the KCA reflection journal)

From this excerpt, we identified that this student reflected on the KCA data with the purpose of (“wondering whether we have really read the notes carefully and applied them to date”) and identified the problem (“the extent of synthesis and collaboration is not higher than our expectation”). This shows interesting insight as the student sees that reference notes and synthesis were important. On the basis of her analysis, the student appeared to reflect on the synthesis of ideas and the quality of ideas, and then took actions to bridge the gap (“creating two group portfolio notes with her group members, one individual portfolio note and several reference notes”, based on the analysis of online discourse in Knowledge Forum). The student not only thought about reading notes and the quality of synthesis notes, but generated useful knowledge building goals including synthesizing ideas and improving ideas collectively. Moreover, she implemented actions to address the gaps, which included synthesizing notes collaboratively and contributing several reference notes. This example suggests that reflective assessment using the KCA helped this student to connect her learning orientation to the knowledge-building goal of synthesis and rise-above.

Promoting idea improvement through reflective assessment using the KCA

Reflective assessment using the Knowledge Connection Analyzer (KCA) appeared to have helped this student focus on the knowledge-building principle of idea improvement. An example of an excerpt from a reflection journal is shown as follows:

I am wondering whether my notes have been improved and the ideas have been developed. According to the KCA data, my notes have been read a lot, but built on less. What is worse, no notes are referred to by others. My notes lack thought-provoking questions. I am not happy with the quality of my notes; it has much room to be improved. I think I can use keywords and refer to others’ notes. (from S.20, the KCA task sheet)

From this excerpt, we identified that this student analyzed the Knowledge Connection Analyzer (KCA) data and identified the problems that were (“my notes have been read a lot, but built on less;” “no notes are referred;” “lack thought-provoking questions”). This shows useful insight as student can now see that the notes were read but not built onto; the students also had the sense that reference notes are important. Based on her analysis, the student appeared to reflect on the quality and importantly she noted and generated further plans to address the problems, which was (“use keywords and refer to others’ notes”). It was encouraging that the student not only thought about signposting (using keywords) but also generated useful knowledge building ideas including making more references to others’ ideas.

Overall, these results suggest that reflective assessment using the KCA appears to have helped students to connect their learning orientation with the important goals of knowledge building, such as improvable ideas, synthesis and rise-above, and community knowledge. In addition, the examples illustrates how having students explicitly engaged in reflective assessment using the KCA may lead them to think more critically about how they were reflecting on the assessment data and their online discourse, and how they were linking assessment data and changes of online discourse.

Discussion and conclusion

Students with low achievement are often disadvantaged in classrooms (Poitras and Lajoie 2013) by a lack of engaging instruction that emphasizes higher-order thinking skills such as collaborative inquiry, metacognition, and knowledge construction (Kung and Linder 2007; Schraw 2007). To counter this problem, knowledge building augmented by reflective assessment is proposed as a promising approach for making schoolwork interesting and accessible to students with low achievement.

In this study, we first investigated whether and how students with low achievement are able to assume the high-level responsibility for collectively accomplishing knowledge building. Second, we explored whether and how students with low achievement were able to carry out reflective assessment using the Knowledge Connection Analyzer (KCA) for knowledge building. We conducted this exploratory study at a school performing in the 10th percentile in government examinations—perhaps one of the most unlikely contexts in which a program that emphasizes higher-order thinking would be expected to succeed.

The findings suggest that students with low achievement in this class are capable of working in progressive inquiry; synthesizing ideas and advancing knowledge-building discourse as a whole. They took responsibility for advancing collective knowledge; they regulated their inquiry as they generated new theories and questions, responded to problems and ideas that emerged as community knowledge, and engaged in productive interactions that expanded community knowledge. They made significant improvement over time from early to later phases, based on quantitative data from the Knowledge Connection Analyzer (KCA) including community connection and references, and qualitative analysis of knowledge building discourse at several levels of granularity; students also used the KCA and companying prompt sheets to engage in reflective assessment in the later phase. In this study, we used changes over time including KCA data and discourse analyses (Time 1, early phase; Time 2, later phase) to show students’ engagement in collective inquiry and progress in knowledge advancement. Although there is no comparison group, the use of change over time to investigate students’ collective inquiry is frequently utilized in studies of knowledge building. For example, in investigating idea improvement in inquiry threads, Zhang et al. (2007) analyzed inquiry threads and rated students’ personal ideas on the continuum from naive to scientific understanding and demonstrated significant progress and improvement over time. Chen et al. (2015), in examining promising ideas, evaluated how young children were able to engage in collective knowledge building through analyzing SNA indices over different phases to illustrate how the design could have supported children’s growth in collective inquiry in knowledge building.

As with previous studies of knowledge building (Chen et al. 2015; Lee et al. 2006; Resendes et al. 2015; Zhang et al. 2007, 2009), we show students’ engagement in collective inquiry and changes over time beyond information sharing to knowledge building. Comparing knowledge building with other online work, this study found that students engaged in sustained discussion, whereas early studies on online discussions report disappointing results regarding deep inquiry (e.g., Hewitt 2005; Hiltz and Goldman 2005). Our study shows that students with low achievement were able to engage in productive interactions and collective discourse growth supported by knowledge building and reflective assessment. This indicates that knowledge building augmented by reflective assessment may have benefitted educationally disadvantaged students who are not typically successful in learning. This finding is consistent with the results of previous research, in which reflective assessment (White and Frederiksen 1998) and collaborative inquiry has been reported to positively influence students with low achievement (Duckworth et al. 2014; So et al. 2010; Zohar and Dori 2003). The findings have important implications for the design of technology-rich environments as metacognitive tools to support students with low achievement, and shed light on how teachers can use these tools to help learners to develop metacognitive skills by engaging in metacognitive practices. In this study, we did not specifically measure variables such as motivation and higher-order thinking skills by giving students questionnaires because they may not tap deep phenomena. Based on the teacher’s and the researchers’ observations, the students were motivated and engaged in their collective work; their knowledge building discourse also reflect their engagement in higher-order thinking that was said to be lacking in students with weak academic abilities.

Regarding the second research objective, we found that students in this class were able to carry out reflective assessment using the KCA, and that reflective assessment using the KCA appeared to have helped students to connect their learning and inquiry with the key knowledge building goals such as community knowledge, collective responsibility for idea improvement, and synthesis and rise-above. The results suggest that developing a knowledge-building environment emphasizing collective efforts and improvable ideas is important and beneficial for students with low achievement and diversity. Knowledge building is enhanced by reflective assessment which might be particularly helpful for students with low achievements. The results also suggest that it is possible that requiring students to be involved in a reflective assessment process facilitated students to think more reflectively about their inquiry tasks and forum writing, and helped students to progressively advance their understanding. To guide students’ use of the Knowledge Connection Analyzer (KCA), we provided students prompt sheets that were both content-related and metacognitive, and corresponded to each of the four questions in the KCA. These prompt sheets aimed not only to help students to make sense of the quantitative data generated by the KCA, but also to encourage students to deploy and develop metacognitive skills. These prompt sheets may have helped students to engaged in a series of metacognitive activities, such as conducting assessment, writing reflections, setting learning and knowledge-building goals, and continuously improving their online discourse. In thinking more deeply about and involving more in reflective assessment, students may have internalized the metacognitive components of monitoring, reflecting, and regulating than they would otherwise. Thus, the students need to continuously reflect on their inquiry and discourse, and we scaffold them to engage in an reflective assessment process of setting knowledge-building goals and reflecting on their progress.

In this knowledge-building environment augmented by reflective assessment, few opportunities were created to facilitate students to reflect collaboratively on the Knowledge Connection Analyzer (KCA) data under the framing of collaborative efforts for data-driven discourse improvement. On the other hand, we also observed some difficulties as students might perceive the assessment data from Knowledge Forum’ integrated tools as individual-based for comparative purpose, and to some extent created a competitive culture. Consequently, some students might still reflect on their own performance when using the KCA data that intended to direct students’ attention on community performance. There were others who appeared to have a better understanding about a community-oriented framework of reflective assessment using the KCA and further work is being conducted.

Knowledge building emphasizes knowledge advancement as the accomplishment of a community, and initial group inquiry such as jigsaw learning, small group discussion, the construction of knowledge wall, knowledge-building talk and classroom discourse were used to build collaborative culture and to frame discourse improvement as a collective responsibility before introducing the KCA. The factors may explain why many students were capable of reflecting on the KCA data from community’s perspective, and may have mediated students’ productive use of reflective assessment. Our findings suggest ways of structuring change in classrooms and even schools, particularly to meet the needs of educationally disadvantaged students. The findings of this study also lay the groundwork for future research on students’ collaborative work and metacognitive activities in relation to the use of data to improve performance.

In the area of knowledge-building research, analyses have mainly relied on online discourses and quantitative server-log data in the Knowledge Forum database. Very few studies have made use of qualitative data—such as interview transcripts, student artifacts, and face-to-face discourse between the teacher and students and between students and students—to characterize the process dynamics that arise from knowledge building. However, to gain a fuller understanding of knowledge building, it is necessary to understand the relationship between online discourse and the nature and dynamics of the practices that develop during knowledge building (Zhang et al. 2007). This study addressed this gap in the literature by capturing the nature and dynamics of the practices that develop during knowledge building.

In addition, the findings of this study highlight the role of teachers in developing student agency and facilitating knowledge building. Unlike many teachers who easily dismiss complex approaches such as inquiry or knowledge building as inappropriate for students with low achievement, the classroom teacher held strong beliefs in the students’ ability, and he actually believed that more innovative approaches are needed to help students with low achievement. The teacher involved in this study established a collaborative knowledge-building culture and norms for idea improvement, as well as helping his students to build their competency of collaboration, inquiry (e.g., questioning and explanation), and reflection (e.g., reflecting on data from Knowledge Forum’ integrated assessment tools, and the quality of and ways to further improve good notes). The teacher’s experience of knowledge building also enabled him to make a valuable contribution to knowledge-building pedagogy. He experimented with new designs of engaging students with low achievement in collaborative knowledge building in class. Classroom observation shows that he regularly engages students in knowledge building talk and productive discourse; helping students to dig deeper and to reflect on their inquiry and Knowledge Forum work. Perhaps most notable was his strong belief that knowledge building is a promising means of motivating and empowering students with low achievement and helping them to develop higher-order thinking, and that students with low achievement are able to take collective responsibility for advancing knowledge in a supportive and appropriate knowledge-building context.

Limitations and future work

The findings of this study indicate that knowledge building augmented with the Knowledge Connection Analyzer (KCA) and its accompanying prompt sheets may have helped students with low achievement to engage in productive reflective assessment—focusing on the knowledge-building goals and thereby collectively improving the discourse created by a community. However, due to the lack of design for collaborative reflection on the KCA data under the framing of data-driven discourse improvement as a collective responsibility, some of the students continued to experience difficulties such as focusing on their own performance rather than that of the whole community. Future research should attempt to solve these problems by implementing and testing a pedagogical design in which use of the KCA is integrated with other aspects of a pedagogical design for knowledge building. Such a design has different components, including (a) establishing a collaborative knowledge-building culture and norms for contribution and participation, (b) regular and opportunistic use of knowledge-building talk that promote collaborative reflection and thoughtful use of the KCA, and (c) framing the data-driven improvement of discourse and reflective assessment as a collective responsibility, for example, designing prompts focusing on collective work (Scardmalia 2002).

The students involved in this study were drawn from a single class taught by one teacher. Therefore, the scope of analysis was restricted by the absence of a comparison group. A comparison as a reference point would have allowed us to determine with more certainty whether the study’s findings could be attributed to the construction of a knowledge-building environment augmented by reflective assessment. We explained earlier that we followed the knowledge building research tradition using multiple methods to show how the design might have impacts on students. Further research would be undertaken to examine the use of reflective assessment with knowledge building. We were also able to draw on a number of video recordings of the teacher’s work in the classroom in previous years, as well as many detailed accounts of knowledge-building discourse in different schools and different classrooms to illuminate the role of reflective assessment on students with low achievement. Nevertheless, further research of this kind would help to illuminate these questions and would help to solve these puzzles that we have.

Conclusion

To conclude, this study has shown that reflective assessment supported by the Knowledge Connection Analyzer (KCA) appears to have facilitated students with low achievement in sustained knowledge-building discourse. Discourse analyses using different unit of analyses over time show that students were engaged in sustained inquiry and collective knowledge advances. Reflective assessment with the KCA asking students to reflect on how they are putting knowledge together, highlights metacognitive components of goal setting, monitoring, planning and reflection that may have helped students to focus on goals and strategies of knowledge building, and thereby facilitate them to advance their knowledge-building discourse.

This study provides an example of the potential of knowledge building augmented by reflective assessment to foster collaborative inquiry and higher-order thinking among students with low achievement in a cultural and educational context that places considerable emphasis on examinations. The study’s findings have practical implications for teachers and researchers who wish to design computer-supported collaborative learning environments or provide instructional support to help students benefit from collaborative inquiry. The study also has theoretical value, as it offers insights into the relationships between reflective assessment, collaborative inquiry, and instructional practice, and the potential affordances of knowledge building for students with low achievement.