Characterizing active learning environments in physics: network analysis of Peer Instruction classroom using ERGMs

Active learning is broadly shown to improve student outcomes as compared with traditional lecture, but more work must be done to distinguish outcomes between different types of active learning. We collected self-reported student social network data at early and late-semester times in a Peer Instruction classroom. The subsequent networks are modeled using exponential random graph models (ERGMs), which are a family of statistical models used with relational data, like social networks. We discuss preliminary ﬁndings using this method for a Peer Instruction class. The best-ﬁt ERGM predicts long “chains” of student edges, such as might arise from students talking along rows in the lecture hall. ERGMs appear to be a promising method for quantifying network topology in active learning classrooms.


I. INTRODUCTION
The Discipline Based Education Research (DBER) community has an abundance of evidence that active learning yields better outcomes than lecture [1].However, there is still much to learn about individual pedagogies that fall under the active learning umbrella and how they impact students, particularly for physics.Previous studies have mostly compared a given curriculum to standard lecture as a way to establish impact, but we need to understand how these pedagogies promote the observed learning gains.By understanding these mechanisms, we as a community can be more deliberate with our planning and curriculum choices.
Active learning, at its core, promotes student interactions in the learning process [1,2].Students are engaging with each other, forming connections that can be translated into a social network.It stands to reason that the structure of the learning environment would manifest in the student social network structure, suggesting that social network analysis can be a valuable tool for characterizing active learning environments.In fact, when comparing the social network of a lecture class with that of a Modeling Instruction class, there were distinct differences [3].Different pedagogies should ultimately give rise to different relationship patterns due to a myriad of factors, such as class size, classroom layout, and particularly how student activities are structured.This work is part of a larger project that aims to characterize active learning pedagogies in physics using classroom observations and network analysis.Here, we aim to use Exponential Random Graph Models (ERGMs) to identify the salient topological features of a social network of students in a Peer Instruction class.The visualization of the chosen networks can be seen in Fig. 1 and Fig.

II. METHODOLOGY
We collected social network data from a Peer Instruction [4] class at two time points early in the term and late in the term.Students were invited to take an electronic survey through Qualtrics during the first week of the term, and again during the 10th week.In this survey, students were first presented with a roster of the names of students in the class, and then were asked to select their name from the list.They were then provided with the class roster and asked to select everyone with whom they had a meaningful interaction with in physics class this week.Of the 113 students enrolled in the course, 105 responded to the early-term survey and 93 responded to the late-term survey.

III. EXPONENTIAL RANDOM GRAPH MODELS
Networks are built on relational data-in this case, the nodes in our network are students and the edges are selfreported connections between students.Several metrics can be used to describe the structure of a network, such as density (the ratio of observed to possible edges), dyad-shared partners, and number of triangles.
Dyad-shared partners is a metric that characterizes the number of shared partners between nodes.A dyad is any pair of nodes.These nodes can be connected or not connected.If they have one partner in common, they are said to be a single dyad-shared partner, or DSP-1.Two shared partners would be DSP-2, three DSP-3 and so on.A triangle is a special case of DSP-1 where the shared partners in question are also connected, making a complete triangle of connections.
These metrics, however, are calculated from an observed network, when there are a myriad of possible networks that could have arisen from the same set of nodes, all of which could have similar or dissimilar metrics to describe the network structure.
The type of curriculum is hypothesized to drive the development of the student network, forming unique structural features.In order to be statistically rigorous in our investigation into the processes that drive these structures to form, we need to consider a set of possible networks and compare them to the observed network.
Network data, however, is inherently relational, so standard regression is not appropriate.Exponential Random Graph Models (ERGMs) can be used to examine how the presence of an edge fits with the parameters measured from an observed network based on structural features, such as the density or number of single dyad-shared partners.The general form of an ERGM tells us the probability of an observed network as a function of the network features we expect to be non-random.The ERGM can be written as: where P (Y = y) is the probability of one specific network occurring, g(y) are the network metrics to fit (like density or number of triangles), θ are the coefficients for said metrics that the ERGM calculates, and k(θ) is the quantity in the numerator summed over all possible network configurations [5].
We can use this fit to simulate a large quantity of similar networks, controlling for similar structures, to see how well the model captures the observed network structure.By comparing an observed network to a subset of possible networks with similar metrics, we are able to establish whether certain structures are a random occurrence or a byproduct of some external factor, such as an interaction typical of the curriculum.For example, presumably, working in groups of three would have an effect on the number of triangles in a network.
ERGMs take multiple inputs for network metrics to fit models to the observed network: metrics such as the number of edges, triangles, dyad shared partners, and more.ERGMs can also include node attributes like gender, age, or major in the fitting of network models (for future work).ERGMs are generative, which means that once a model is fit to an observed network, the model can be used to simulate large numbers of networks.This allows us to determine if an observed network parameter is typical or extreme.

IV. ANALYSIS
For the Peer Instruction network, we observe chain-like structures seen in Fig. 1 and Fig. 2. In this network, a chain of three students forms a non-connected DSP-1, where the central student is the shared partner to the outer student dyad.Since we observe chain-like structures in our networks, which in turn suggests a dependence on DSP-1, we chose to use this as a fit parameter in the ERGMs.We also observe zero triangles in both the early and late-term networks, so we also include a model to fit the number of triangles in our preliminary exploration.From these observations, we chose to test three ERGMs: • Edges only model.This fit assumes no structure, each edge is equally likely.This is the null hypothesis.• Edges + triangles.Fits number of edges and also tries to match zero triangles condition of the observed networks.• Edges + DSP-1.Fits number of edges and also tries to match 227 or 289 DSP-1 condition of the observed networks.We compare numerical metrics of each model, including the fit coefficients, the standard error, the AIC, the BIC, and the probability of an edge in Table I.The coefficients in an ERGM represent the change in the log-odds likelihood of an edge for a unit change in a predictor variable.So, for the edges-only model, a unit change would be ±1 edge.For the DSP-1 model, the unit change would be ±1 DSP-1.
We can calculate the probability of edge formation via where the log-odds is calculated from the coefficients via log-odds = edge coefficient where the other coefficient and new metric refer to either triangles or DSP-1 [6].
The edges + triangles model is not able to establish a fit for either observed network, due to the existence of zero triangles in the observed networks, as seen with the NaN AIC and BIC and the infinite fit coefficients.Due to the lack of convergence with the edges + triangle models, we restrict further discussion to the edges only and edges + DSP-1 models.
For the edges only model, the probability of an edge correlates directly with the density of the network.This assumes that each new edge is equally likely.However, with the edges + DSP-1 model, if a new edge will create a new DSP-1, it has a higher probability of occurring than a random connection.This probability is even higher in the late-term network than in the early-term network.This leads us to believe that DSP-1 will be a significant feature of these Peer Instruction networks.
Once the models were created, we simulated 100 networks for each model for early/late-term networks.In Fig. 3, we plot the number of DSP-1 for each simulated network generated from the edges only models for both early and lateterm networks.The red line indicates the observed values of DSP-1.By inspection, the observed networks both have a lower count of DSP-1 than expected by random distribution of edges.The early-term network has 227 DSP-1, and the late-term network has 289 DSP-1.II: Fit coefficients from ERGM models.For multi-line models, the first value is the edge coefficient and the second value is the additional fit parameter coefficient.The reduction in AIC/BIC of > 10 points is strong evidence that the edges + DSP-1 model is a better fit than the edges-only model [7].Using the edges + DSP-1 ERGM models, we also plot number of DSP-1 of the simulated networks in Fig. 4. By including DSP-1 as a fit parameter, the model more accurately predicts the observed values of DSP-1, suggesting that DSP-1 is a significant feature of the observed Peer Instruction networks.This visual inspection corroborates the lower AIC and BIC as shown in Table I.

Number
We can also see how each model converges with respect to number of observed triangles.As stated previously, both the early and late-term networks had zero observed triangles.For a completely random distribution of edges, the zero triangle condition is less likely than 1 or 2 triangles, as seen in Fig. 5a.The edges + DSP-1 model increases the likelihood of the zero-triangle condition, seen in Fig. 5b.Again, this corroborates the lower AIC and BIC for the edges + DSP-1 model in Table I.We have used three ERGM models to fit early and late-term Peer Instruction networks and showed that the edges + DSP-1 model more accurately represents the observed networks than the edges-only or edges + triangle models.This leads us to believe that ERGMs are a promising tool for characterizing different active learning pedagogies.

Number of triangles
For the data shown here, the dependence on DSP-1 is a good example of how pedagogy can possibly manifest in a network.In the class that gave rise to these particular networks, students sit in rows in a large lecture hall, so they are constrained to interacting with people in their immediate vicinity.While we cannot definitively say that the chain-like structures seen in Figs. 1 and 2 are due to the pedagogy, the emergence of such structures is consistent with the geography of the classroom and the instructions for interaction typical of a Peer Instruction curriculum, such as "discuss with your neighbor." Of the three ERGM models tested, we determined that DSP-1 was the best predictor for these Peer Instruction networks compared to the other two predictors, edges and triangles.As such, we are hopeful that ERGMs will also be able to identify the significant predictors for networks using different pedagogies.From there, we can begin comparing ERGM coefficients within pedagogies and across pedagogies to see if these values are unique to a specific curriculum.We are hopeful that ERGMs will be a successful tool to characterize pedagogical differences.

VI. CONTINUING WORK
Network data has been collected for six active learning curricula.Continuing work will include the analysis of each of these curricula using ERGMs.We would like to test each ERGM model with different data sets within the same curriculum to see how well they describe an individual network, and then we would like to compare ERGM models across different curricula to see if they are distinguishable between pedagogies.
Is there a set of coefficients that is more likely to arise for a Peer Instruction network over a Modeling Instruction network?Do these coefficients remain somewhat constant across different observations of a Peer Instruction network?Do these coefficients correlate to the amount of time spent lecturing in the classroom?Future work will unpack these questions and hopefully allow us to quantitatively characterize our selected active learning pedagogies.
FIG. 3:We simulated 100 networks using the edges only ERGM models and plotted the number of DSP-1.The red line is the number of DSP-1 in the observed Peer Instruction networks.

FIG. 5 :
FIG. 5: Number of triangles formed with 100 simulations of edges-only ERGM model compared to edges + DSP-1 ERGM model-Late term network V. DISCUSSION