Cognitive discourse during a group quiz activity in a blended learning organic chemistry course

2.1 Blended learning environments

In blended or flipped learning environments, lectures are replaced by short videos and readings, and class time is used for problem-solving and collaborative group activities (Seery, 2015). Blended environments allow students to be self-paced learners and facilitate skill-building during the synchronous class period. Research on flipped organic chemistry courses has shown that students achieve at higher rates in this environment (Flynn, 2015; Liu et al., 2018; Mooring et al., 2016; Shattuck, 2016). Mooring, Mitchell, and Burrows also found that students in a flipped organic chemistry course had more positive attitudes toward chemistry (2016). Specifically, students in the course were more emotionally satisfied and felt that the content was more intellectually accessible than students in the traditional course. The organization of course materials and exposure to guided problem-solving has often impacted students with identities underrepresented in STEM (Theobald et al., 2020). Despite this knowledge, the research literature does not fully address the factors that drive students’ success in such courses. To expand such knowledge, herein, we present an analysis of student dialogue during a collaborative quiz administered as a component of a blended learning environment.

2.2 Collaborative testing in higher education

Collaborative testing is supported by social interdependence theory (Johnson et al., 1989, 2007), which is under the umbrella of social constructivism. In this theory, the group’s outcomes depend on students working together to achieve common goals. Therefore, the interactions among group members lead to co-constructed knowledge (Bloom, 2009; Zhao & Kuh, 2004). Like blended learning environments, collaborative learning activities enhance students’ beliefs in their agency as learners (Mahoney & Harris-Reeves, 2019). Collaborative testing is a student-centered, active learning approach where two or more students discuss the content of a written assessment (i.e., an exam or quiz) before submitting individual or group solutions. The rationale for the implementation of such activities not only includes reduced test anxiety (Bloom, 2009; Kapitanoff, 2009), but also has the added benefit of improving students’ interpersonal skills (Dallmer, 2004; Scager et al., 2016). Similar outcomes have been demonstrated in STEM and non-STEM courses (Russo & Warren, 1999).

In a controlled study, undergraduate students in a collaborative testing environment performed better (Gilley & Clarkston, 2014) when comparing pre- and post-outcomes on individual assessments. Other studies have shown that group exams resulted in increased learning and, more impressively, higher scores on the exam when readministered, signaling knowledge retention (Bloom, 2009; Cortright et al., 2003). A more recent study of group assessments in a statistics course has shown that scores on exams taken as a group were significantly higher than on exams taken individually (Kapitanoff & Pandey, 2018). The result was more pronounced for students with lower GPAs and higher test anxiety. Arguably, low-performing students or those without the knowledge needed to complete the assessment would benefit from group engagement that promotes understanding of the concepts. High-performing students might benefit from having their knowledge confirmed. Therefore, this notion of benefit needs to be better defined. Further, some studies have indicated no difference in test scores resulting from student engagement in collaborative testing (Gilley & Clarkston, 2014; Giuliodori et al., 2008; Harton et al., 2002; Johnson et al., 2007; Leight et al., 2012; Springer et al., 1999). These contradicting results suggest that there is still much to understand about the conditions under which group testing lead to increased learning.

A study by Mahoney and Harris-Reeves attempted to address this gap in the literature by exploring the effect of collaborative testing on higher-order thinking skills of undergraduate students enrolled in a sports and exercise psychology course in Australia (2019). Three examinations were part of the course; however, only the second examination was delivered independently and collaboratively. Students had 45 min to complete the multiple-choice exam independently during the testing scenario. The students were then given 30 min to complete the same assessment in groups. All students taking the collaborative test did better overall on the higher-order questions than those who took the independent tests. The result was observed for all learners regardless of their typical level of performance. While the authors noted that the performance could be related to the discussion and deliberation of the exam content, this engagement was not characterized. The authors remarked that not having this information limited their ability to address: (1) how interactions among the students lead to better answers to higher-order questions and (2) the quality of the interaction between higher and lower performers.

3.1 Community of inquiry (CoI)

The CoI framework is based on a constructivist approach to learning (Swan et al., 2009) that posits that learning occurs when the learner is actively involved in creating meaning and constructing knowledge. CoI accomplishes this by facilitating a rich educational experience (Garrison, 2013) that emphasizes the interrelationships between teaching, cognitive, and social presences and the collective influence of the presences on the student learning experience, Figure 1 (Garrison, 2019). In this way, the framework informs critical elements needed for learning and has been used to characterize student engagement in online and in-person components of the blending learning format (Garrison, 2013, 2019; Swan et al., 2009).

Figure 1:

Community of inquiry framework.

Studies have found a relationship between students’ retention in science and their positive social presence, the learner’s ability to inject their personality into the learning environment and learn through engagement with others in the environment (Akyol et al., 2009; Arbaugh & Benbunan-Finch, 2006; Richardson & Swan, 2003). Social presence has impacted students’ motivation and participation, actual and perceived learning, and course and instructor satisfaction. Also, participants actively contributed when students enjoyed the group dynamics (Garrison, 2019).

Cognitive presence can significantly influence the teaching and social presences (Molinillo et al., 2018) and is, therefore, critical to the function of the CoI as it informs how students and instructors co-create meaning and confirm understanding in a learning environment. The teaching presence includes direct instruction and instructional management. This component of the CoI also includes selecting, organizing, and presenting course content and developing learning activities (Rahim, 2022).

Adding to the use of the framework described above, CoI shaped the design and structure of the Organic Chemistry course that is the focus of this study. The course introduces concepts through online lecture videos and required readings (Fullilove et al., 2017; Sanders-Johnson et al., 2020; Sanders-Johnson et al., 2021; Sanders et al., 2019; Winfield et al., 2019). Collaborative and active learning strategies are prominent features of the course, along with instructor-created content and learning strategies. The current study is primarily situated at the intersection of the cognitive and social presences in that we are examining student discourse during group activities.

3.2 The interactive, constructive, active, passive (ICAP) framework

The ICAP framework is a taxonomy that characterizes students’ overt behaviors during learning activities. The framework categorizes learning behaviors as interactive, passive, active, or constructive. It hypothesizes that students become more engaged in learning as they progress from passive to interactive, Figure 2 (Chi & Menekse, 2015; Chi & Wylie, 2014). In passive engagement, the student receives information but contributes little or superficially to the discussion. For this study, passive engagement is further divided into negative passive and active passive, according to work by Chen (2018). Negative passive refers to instances in which the student indicates disagreement/agreement or understanding/confusion without further elaboration (Chi & Menekse, 2015; Chi & Wylie, 2014). Active passive dialogue, in contrast, occurs when a student demonstrates a level of listening that enables them to describe or summarize what another student stated. In this case, no new information is added to the dialogue. Passive, whether active or negative, results in shallow understanding. However, passive engagement does not necessarily constitute disengagement or off-task behaviors. When students actively engage in the learning process, they can cycle through each phase of the ICAP model.

Figure 2:

Interactive, constructive, active, passive (ICAP) taxonomy for overt learning activities. Each level demonstrates students’ engagement at a given moment in the discussion.

During active engagement, no new information is added by the student. However, students are physically involved in something related to the learning activity (Chi & Menekse, 2015; Chi & Wylie, 2014). Some examples of active engagement during discourse include reciting a memory line, reading the problem aloud, asking a question for clarification, and repeating what is heard. Constructive engagement involves a higher level of cognition and produces new information relevant to the activity (i.e., textbook, problem set, or verbal and written instructions). Such instances include self-explaining, creating a concept map, and filling-in knowledge gaps while dialoguing. Therefore, an instance of constructive engagement reflects a student’s reasoning. This level of independent knowledge generation is associated with greater understanding and learning gains.

Finally, interactive engagement is evident in a student’s contribution to new information in response to another peer’s input (Chi & Menekse, 2015; Chi & Wylie, 2014). An instance of dialogue can be interactive only if it directly responds to a comment from another student in the group. Otherwise, that dialogue is constructive. Interactive engagement involves the co-creation of knowledge with peers. In this case, the student makes a substantive contribution to the discussion through new lines of thinking, deliberation, and feedback. Interactive dialogue leads to the deepest level of understanding since the group gains new understanding based on their collective contribution of new ideas.

In chemistry, researchers have used the ICAP framework to qualify engagement during various learning activities. For example, El-Mansy used the framework to examine peer-peer interactions in general chemistry, finding that cognitive engagement was often impaired by students’ flawed understanding of the concepts (2022). In a physical chemistry course, the framework allowed faculty to define a correlation between faculty facilitation strategies and the quality of group engagement (Liyanage et al., 2021). The ICAP taxonomy was used in conjunction with Bloom’s taxonomy and cognitive load theory to illustrate that an interactive desktop tool leads to a deeper understanding of fluid dynamics in a chemical engineering course (Kaiphanliam et al., 2021). In the study, researchers created activities that were purposefully passive (i.e., students constructing a basic pump) or interactive (i.e., students completing a desktop activity and discussing outcomes in a group). In the latter case, students were provided with a worksheet to prompt interactive or constructive engagement, but the extent to which students demonstrated either was not addressed.

Similarly, biochemistry instructors applied the ICAP framework to create impactful learning experiences (Hilton et al., 2022). The authors have employed PeerWise in a biochemistry course to encourage constructive and interactive engagement by authoring and answering questions. Their work found that the exercise promoted cognitive gains and engagement at the higher levels of the ICAP framework. However, the findings did not provide insight into the characteristics of that engagement. In an essay by Linda Hodges, the author evaluates utilizing the ICAP framework for improving the implementation of the learning strategy in science classrooms (Hodges, 2018). Hodges highlights work by Sandi-Urena et al., which shows a correlation between students’ ability to explain their choices in a chemistry laboratory and their metacognitive abilities. In follow-up work, Hodges emphasizes the alignment of constructive engagement to higher levels of Bloom’s taxonomy (Leupen et al., 2020).

Each study described illustrates the use of the ICAP framework to understand the level of cognitive engagement students demonstrate during learning activities. The studies quantify students’ overt physical and verbal behaviors at specific intervals. Specifically, Chi and Wylie (2014) and later Chi and Menekse (2015) utilized the framework to analyze conversations during active learning. Likewise, the study described herein utilizes ICAP functions as a theoretical and methodological framework. When applied as a theoretical framework, it explains the underlying cognitive processes related to students’ overt behaviors as a function of their dialogue patterns. As a methodological framework, it allows us to apply the four levels of engagement in coding students’ overt verbal behavior during the group activity, Figure 2 (Chi & Menekse, 2015).

5.1 Setting and participants

Spelman College is a fully accredited baccalaureate institution historically established for women of African descent. The College has a total enrollment of 2100 students, all of whom identify as Black women. Less than 10 % of these students major in the natural sciences and mathematics (Office of Institutional Effectiveness, 2022). Nevertheless, the College is the top producer of Black women graduates who earn doctorates in these fields (Hrabowski & Henderson, 2021). The study includes observational data from three unique groups completing two different group quizzes. The current study was implemented in a second year, college-level course (age group 18–20) in the United States. The course spans a 16-week semester with three 50-min weekly in-person meetings. All students enrolled in the first-semester organic chemistry lecture in Fall 2017 were required to participate in the group quizzes described herein. The study was approved by the IRB boards of the associated institutions.

5.2 Course structure

Course content and resources were available on the learning management system, which included lecture videos and pre- and post-assessments students completed . Pre- and post-assessments were administered for each topic and contained 10–15 multiple-choice questions varying in difficulty. To reinforce online content, the instructor reviewed the main points from the topic during the first 15–20 min of the in-person class period. The instructor also used this time to answer questions about the previous class period. Students used the think-pair-share model (Kaddoura, 2013; Marzano & Pickering, 2005) to complete the problem set during the remainder of the period (20–30 min). Problems not finished in a given class period were completed as homework, used to start the discussion during the following class, or selected for random grading. A post-quiz was administered at the end of a topic to assess students’ concept mastery.

5.3 Data collection

Regarding the physical classroom workspace, student groups were situated around rectangular tables configured to allow students to communicate face-to-face. Each group had a dedicated camera, short-throw projector, whiteboard, and iPad. The cameras were set at fixed angles allowing the groups to be video and audio recorded. iPads, equipped with the screen-capturing app Explain Everything (Explain everything for IOS devices), were used to record group responses to the quiz prompts. We only analyzed recordings where all group members could be seen and heard on camera. Therefore, observational data for groups A and B were analyzed for quiz 2, and observational data for groups B and C were analyzed for quiz 3. The pseudonyms for students in the selected groups are shown in Table 1.

5.4 Data analysis

The recordings were transcribed, and the transcripts were separated into excerpts. Each excerpt ranged in length and consisted of a single group’s response to one quiz prompt. Each excerpt was then divided into speaking turns, with each line representing when a different student began speaking. Since this is a group discussion, it is expected that student discourse may vary throughout the discussion in response to other group members as the conversation evolves. Characterizing engagement based on speaking turns allows us to take a holistic view using natural breaks in the discussion. On the other hand, coding based on timed intervals could inhibit the meaning of a student’s statement if the designated time ends while the student is speaking and therefore may not accurately capture a student’s level of cognitive engagement. Each speaking turn was coded as interactive, constructive, active, active passive, or negative passive using the definitions indicated by the ICAP framework, Figure 2. Speaking turns unrelated to the group quiz were coded as “off-task.”

After the first round of collaborative coding by authors SG and SRM, a third individual coded a sample of transcripts to establish interrater reliability. This coder used the codebook to analyze two sections of the transcript from quizzes 2 and 3. Disagreements were discussed, and the codes were adjusted or refined accordingly. Coding continued until satisfactory interrater reliability was reached between coders. Cohen’s kappa for interrater reliability was calculated as 0.78, which indicates a good agreement between coders (Watts & Finkenstaedt-Quinn, 2021). The rest of the data was then coded by SG.

The percent of each type of dialog (interactive, constructive, active, or passive) present was calculated as the number of each type of dialog out of the total number of all dialog types. For example, if there were 20 speaking turns in the excerpt and 10 were coded as active, that excerpt is 50 % active dialogue. The count for interactive, construction, active, or passive codes were also determined for each student in each group analyzed. Off-task dialogue was not included in the total code count.  In this way, dialog patterns were determined for periods when students were intentionally engaged in the learning activity.

5.5 Interaction quality

The interaction quality of each excerpt was classified as high, medium, or low using a modification of a scheme by Chi, see Supplementary information (Menekse & Chi, 2019), in conjunction with the ICAP percentages and other factors (i.e., the number of group members engaged in dialogue). Dialogues were coded overall as low interaction if the dialogue is mainly passive or active, has little input from group members, or if one student suggests a solution that the others agree to without discussion. Medium interaction occurred when the discussion was primarily active. In this scenario, one student provides most of the ideas, and the dialogue among the group is discontinuous; that is, students’ speaking turns were not in response or related to the previous idea. An excerpt was coded as high interaction if there were a substantive number of student statements that were constructive and interactive, and most students in the group contributed to the discussion.

5.6 Characterizing the cognitive level of quiz prompts

A critical feature in the group learning activities is the design of the quiz prompts used to facilitate discussion. Therefore, Marzano’s Taxonomy was used to define the cognitive level of each prompt in the group quizzes (2001). The levels from lowest to highest are retrieval, comprehension, analysis, and knowledge utilization, Figure 3.

Figure 3:

Definition of Marzano levels.

6.1 Outcomes for research question 1: what is the observed interaction quality when students participate in a group quiz activity?

The dialogue excerpts provided in Tables 2 –5 illustrate the dialogue observed among the groups that completed quizzes 2 and 3. Below, we assess the interaction quality of the excerpts. These cases are presented as a comparison between groups working on the same quiz prompt in order to highlight the characteristic differences in interactional quality. That is, we present cases of low versus medium interaction quality and medium versus high interaction quality.

Tables 2 and 3 compare low to medium interaction quality on the same prompt. These excerpts are taken from the discussion of quiz 2, prompt 2: Draw a template to illustrate the anti-conformation of an alkane.” For this prompt, students had to choose any alkane and draw a Newman projection in the anti-conformation.

For group B, the discussion was concise, consisting of only four speaking turns, Table 2. There was no interactive discussion of ideas. In this excerpt, one student, Farrah, offered a solution (turn 2), which is copied to the iPad as the group’s final answer. No one asked for further explanation of how Farrah arrived at her answer. Although the group provided a reasonable answer to the prompt (Figure 4), the lack of interactive or constructive dialogue likely prevented the group members from gaining additional understanding of these concepts. The observation is similar to findings by Chi that assert that non-contributing partners may have little meaningful gains from the discussion (Chi & Menekse, 2015). It is also possible that the prompt did not encourage deeper interogation of the concept; therefore, the group did not think any further discussion was necessary (Leupen et al., 2020).

Figure 4:

Group B’s final answer to quiz 2, prompt 2, “Draw a template to illustrate the anti conformation of an alkane”.

In contrast to group B, we qualify group A’s response to the same prompt (Table 3) as medium interaction since the dialogue is primarily active and is 64 % of the dialogue; however, all group members participated in the discussion. Also, students’ responses are discontinuous because each student makes assertions independent of the other. That is, “why” or “how” questions are absent, and the students are not expanding each other’s comments. Group A drew a cyclohexane chair in equatorial and axial conformations instead of a Newman projection (Figure 5). Candice mentions Newman projections twice (turns two and 6). However, the group did not address her question and apparent confusion. She did not have an opportunity to further discuss her ideas, which may have led the group in a different direction. Perhaps, this prompt triggered the students to remember the idea of conformations in general since both Newman projections and chair structure are representations of conformations, the former for alkanes and the latter for cycloalkanes. The students seemed to conflate these two types of conformations.

Figure 5:

Group A’s answer to quiz 2, prompt 2, "Draw a template to illustrate the anti conformation of an alkane."

According to Chi’s assertions through research studies using the ICAP framework, the excerpts from group A and B (Tables 2 and 3) on this prompt primarily falls within the passive and active dialogue categories that lead to shallow learning outcomes and limited learning gains. When comparing the two excerpts, it is interesting that the quality of dialogue does not necessarily lead to a correct answer. Although group A (Table 3) qualifies as medium interaction dialogue, their final answer is considered incorrect. Also, group B had a correct answer, but the dialogue was minimal (Table 2).

Tables 4 and 5, describe cases of medium and high interaction dialogue, respectively. Here, the dialogue for groups B and C are compared for quiz 3 prompt 1c: “Describe the relationship between bond length, pK_a, leaving group ability, and reactivity.” To respond to this prompt, students had to understand the meaning of each term and determine how the concepts were related.

Group B’s dialogue reflects high interaction, Table 4, and group C’s conversation reflects medium interaction, Table 5. There was a significant difference in the percentage of interactive and constructive statements between the groups. Constructive and interactive talk accounted for 45 % of group B’s conversation and only 20 % of group C’s conversation for the same prompt.

Group B’s dialogue for this prompt qualifies as high interaction since all the students engaged in the dialogue and built upon each other’s statements. In fact, every student in this group contributed at least twice to the discussion. Students in the group asked each other questions as they co-constructed their ideas to come to a mutual answer. Group B had long sections of constructive or interactive dialogue (Table 4, turns –11 –22), particularly between Brittany and Farrah. The students worked together to make sense of the relationship between bond length, pK_a, leaving group ability, and reactivity. Farrah asked questions for clarity several times throughout the problem (turns 11, 22, 26). Farrah’s questions helped to move the group forward in their thinking. Also, Dawn helped the group to formulate the answer to this problem by contributing interactive dialogue in response to Brittany and Farrah’s statements (turn 15). Brittany then organized and repeated the information for the group’s scribe (turn 21). The group summarized their conclusions in turns 29–32. Their final written answer is shown in Figure 6. This group discussed several aspects of the prompt and concluded that shorter bonds are stronger (turns 4 to 9) and that a lower pKa is a better leaving group (turn14). Although Brittany’s contribution that shorter bonds are stronger acids was incorrect, the group had a rich discussion on all aspects of the prompt and the relationship between these concepts.

Figure 6:

Group B’s answer for quiz 3, prompt 1c, “Based on the data from the table completed for problem 1, describe the relationship between bond-length, pKa, leaving group ability, and reactivity.”

In contrast, group C primarily used active dialogue to discuss the solution (Table 5) and, therefore, qualifies as medium interaction. Morgan proposed some initial reasoning to explain the relationship between bond length and strength (turn 6). However, after Morgan’s input, the constructive or interactive dialogue did not continue and the group did not examine why this was the solution. Instead, the group summarized Morgan’s initial answer to form a response, Figure 7. Although, the final answer was more accurate than group B’s, the dialogue in group C may not have led to enhanced learning outcomes among all the group members based on the ICAP framework.

Figure 7:

Group C’s answer for quiz 3, prompt 1c, “Based on the data from the table completed for problem 1, describe the relationship between bond-length, pKa, leaving group ability, and reactivity”.

Individual Contributions to Group Dialogue or quiz 3, prompt 1c. An additional analysis of the individual contributions to the dialogue for quiz 3, prompt 1c, was conducted. Note that active passive (AP) and negative passive (NP) are combined into just Passive (P) in Figure 8. From this analysis, we can observe that every student in group B contributed to the group’s dialogue on this prompt. Although the number of contributions by each group member is different, each student in the group participated in either constructive or interactive talk or both constructive and interactive talk during this discussion. For group C, we observed that one group member, Michelle, did not participate in the discussion and only two students, Nicole and Morgan, participated in constructive or interactive dialogue for group C. This more detailed analysis is another indicator of the quality of the group’s dialogue and the implied differences in learning opportunities between groups B and C.

Figure 8:

Individual contributions to dialogue quiz 3, prompt 1c for (A) group B and (B) group C.

6.2 Outcomes for research question 2: What is the relationship between the cognitive level of the quiz prompt and the ICAP level of dialogue?

Given the variability in the interactive quality of the groups’ dialogue, we wanted to examine if the cognitive level of the prompt, defined by Marzano’s taxonomy, influenced dialogue quality. Each prompt was assigned a Marzano level, Table 6.

There was an increasing amount of constructive and interactive dialogue as you move from prompt 1 through prompt 6 for quiz 2. Prompt 1 was at Marzano level 1. Prompts 2 and 3 were Marzano level 2. Prompts 4, 5, and 6 were at Marzano level 3. We observed that the level of constructive and interactive dialogue was highest and most consistent when prompts were at Marzano level 3, Figure 9. In fact, for quiz 3, all the prompts are at a Marzano level 2 or 3. Chi-squared statistical analysis was used to compare the differences in high-level dialogue (constructive and interactive) and low-level dialogue (passive and active) between Marzano level 2 and 3 prompts. The chi-squared analysis was statistically significant at p < 0.05 (chi-square statistic = 4.34; p = 0.037).

Figure 9:

Marzano level and cognitive engagement. Percentages of ICAP categories versus Marzano level for (A) quiz 2 and (B) quiz 3.

Marzano level 3 prompts required students to compare and contrast concepts (prompts 4 and 5) or extend their understanding of concepts by developing an analogy (prompt 6). In comparison to Marzano levels 1 and 2, level 3 prompts were more likely to provoke students to ask questions and engage in interactive and constructive dialogue. The level 3 Marzano prompt also promotes higher interaction quality, suggesting that students collaborate and build on each other’s statements and responses. Higher interactional quality also indicates that students ask more “why” and “how” questions. This type of interaction typically leads to deeper learning among all group members (Chi & Menekse, 2015).