The Varieties of Individual Engagement (VIE) Scales: Confirmatory Factor Analyses across Two Samples and Contexts

The field of public engagement, participation and deliberation is fraught with conflicting results that are difficult to interpret due to the very different methods and measures used. Theory advancement and consistent operationalization and assessment of key public deliberation and engagement variables will benefit considerably from standardized measures of constructs and the ability to compare across studies. In this article, drawing from social and educational psychology, we describe the theoretical bases for scales assessing eight varieties of participant engagement that may be experienced during participation activities: Active learning, conscientious, uninterested, creative, open-minded, closed-minded, angry, and social engagement. We describe our development of scales to measure these varieties of engagement, and results from three confirmatory factor analyses across two very different populations (college students and city residents) and three different engagement activities (reading background information, deliberating about ethical scenarios, completing an online survey). Finally, we examine evidence of the convergent and divergent validity of the scales by examining their relationships with each other and theoretically-relevant individual and situational characteristics. Findings indicate the scales have good psychometric properties and show evidence of construct validity. We discuss how these scales might be used in reflective practice and research, and identify questions that public engagement researchers and practitioners will find useful in their work.


INTRODUCTION
Although numerous studies have been conducted to examine the effects of deliberative public engagements on important and desired outcomes, the field is still fraught with conflicting results (Delli Carpini, Cook, & Jacobs, 2004). These inconsistencies undoubtedly confuse researchers and practitioners about when and why certain engagement strategies will result in some outcomes and not others, and they make it difficult for practitioners to know which engagement methods are most likely to achieve their specific purposes. One approach to beginning to understand such conflicting results, and to help practitioners gain a more nuanced understanding of the impacts of their design choices, is to examine potential mediators of positive and negative outcomes. For example, examining the cognitive, affective, and behavioral responses of persons involved in such engagements might reveal clues about why different public engagement methods result in different outcomes. However, in order to compare results across studies, there is a need for-yet a lack of-psychometrically valid measures of such potential mediators. 1 The purposes of this article are, first, to propose a set of measures of individual engagement likely important to public engagement contexts; and second, to explore the utility of such measures. To achieve the first aim, we present theoretical background and discuss why certain varieties of individual engagement may be of interest to public engagement practitioners and researchers. To achieve the second aim, we present preliminary evidence for the structural, psychometric, convergent/divergent, and construct validity of scales designed to assess the proposed varieties of engagement. Specifically, we report results of data from a sample of college students reading and then deliberating about the future development and regulation of nanotechnology and a sample of community residents deliberating about city budget choices. By examining the scales across different samples and situations, we are able to assess the stability and generalizability of our hypothesized measurement model. We also present evidence for the reliability and convergent, divergent, and outcome validity of the scales, as well as directions for future research and measure development.

BACKGROUND AND THEORY
Our framework for studying public engagement at the individual level starts with assuming five broad categories of important variables, as shown in 1 The lack of high-quality measures is actually a much broader problem within the area of public engagement. As noted by Rowe and colleagues, "even in empirical evaluations that detail and justify the evaluation criteria used, instrument development is rarely discussed, and neither is the issue of instrument quality" (Rowe, Horlick-Jones, Walls, Poortinga, & Pidgeon, 2008, p. 421). Figure 1. 2 The first category of variables (element A) includes characteristics of public engagements and how they are designed (e.g., modes of interaction, purposes of the engagement, different forms of discussion or decision rules, presence or absence of experts, and so on) (see Rowe & Frewer, 2005, for a review). At the other end of the public engagement process are the often-cited, potentially beneficial, individual-level outcomes (element E) such as knowledge, changes in attitudes toward the topics discussed, and increases in democratic values (see, e.g., Rowe & Frewer, 2004). Drawing from common psychological theories (e.g., Bandura, 1999;Mischel, 2004), our framework recognizes that public engagement features and characteristics do not directly impact outcomes, but instead are filtered through individuals' perceptions (element C) which are likely to be impacted by personal characteristics (element B) which also may directly impact how participants engage and moderate other effects. In between perceptions and outcomes, however, are important, transient, cognitive, affective and behavioral "states" that characterize individual participant engagement in the event (element D). These states are the focus of this article. 2 Although we focus at the individual level, we acknowledge that public engagement could be studied at other levels, and that it is important to attend to the level of analysis (e.g., individual, group, society), consider the level at which different mechanisms impact outcomes (see, e.g., Wang & Gordon, 2011), and distinguish individual (micro) and environmental (macro) level processes (see, e.g., Kim & Tadisina, 2007;Kozlowski & Klein, 2000). For example, at the event level, one might compare different classes of events for their impacts on policies; or at the societal level, one might examine the effectiveness of different policies that have been developed as a result of different public engagement techniques. Within the field of public engagement, participation, and deliberation, 3 there is some acknowledgement that these states matter. For example, Rowe and Frewer (2004) discuss the importance of acceptance and process criteria, and stress that such criteria need to take into account (and presumably avoid) participant confusion, information overload, social loafing, and so on, implying that how participants engage is ultimately important. In other often more general contexts, researchers have examined why citizens varyingly approach political issues with enthusiasm, aversion, with their minds resolutely made up, or with more open-minded, deliberative, or tolerant responses (Haas & Cunningham, 2013;MacKuen, Marcus, Neuman, & Miller, 2010;MacKuen, Wolak, Keele, & Marcus, 2010). That research has focused especially on the emotional factors that may impact individual engagement with policy or political issues.
By comparison, however, individual engagement has been more explicitly and expansively examined in the educational psychology literature than in the field of public engagement (see Fredricks, Blumenfeld, & Paris, 2004 for a review). In educational psychology, engagement has been defined as the "behavioral intensity and emotional quality of a person's active involvement" in a task or set of activities (Reeve, Jang, Carrell, Jeon, & Barch, 2004, p. 147). Others further note that engagement is a multifaceted construct that includes not only behavioral and affective but also cognitive components (Fredricks et al., 2004;Linnenbrink & Pintrich, 2003). Drawing heavily from the educational psychology literature, in the next sections we describe several forms of individual engagement. While these may not be the only forms of engagement that matter, we argue that they are some of the forms especially likely to matter in the context of public engagements, especially deliberative engagements.

Active Learning and Metacognitive Engagement
To the extent that engagement practitioners are interested in increasing public knowledge and understanding (e.g., of complex topics such as municipal budgeting or science), active learning and metacognitive forms of engagement are likely to be important. Engagement characterized by active learning and metacognition has been an important part of numerous educational theories, including theories of self-regulated learning, information processing, and learning styles (Martin, Watson, & Wan, 2000;Pintrich, 2004;Vermunt & Vermetten, 2004). Self-regulated, metacognitive activities include active learning strategies that promote "deep" rather than "surface" cognitive processing (Biggs, 1979;Chin & Brown, 2000;Dinsmore & Alexander, 2012). Such strategies include transformation of information by reorganization, simplification, or elaboration, rather than more passive information intake (e.g., passive listening or rote memorization). Research has found that active learning engagement is affected by individual differences such as intrinsic motivation and need for cognition (Cacioppo, Petty, Feinstein, & Jarvis, 1996;Nussbaum, 2005); as well as by situational factors and the manner in which learning activities are designed (Kauffman, Zhao, & Yang, 2011;Peters & Kitsantas, 2010). Deliberation practitioners may be interested in assessing whether their methods facilitate active/metacognitive engagement because deep processing has been found to relate to learning gains, knowledge transfer into new contexts, and the assessment and refinement of ideas (Chin & Brown, 2000;Prince, 2004). Thus, active engagement also may be important for encouraging deliberation participants to see connections between subject content and policy.

Conscientious Engagement
Practitioners may wish to know if their methods resulted in conscientious engagement because deliberation often emphasizes effortful and careful weighing of evidence, consideration of multiple arguments, and disciplined attention to detail. As with active engagement, the tendency to engage conscientiously likely varies between individuals. Trait conscientiousness refers to the general tendency to be responsible, careful, thorough, organized, efficient, and trustworthy (Costa & McCrae, 1992), and has been related to use of thinking styles characterized by preferences for structure and guidelines, focusing on one thing at a time, and working independently on concrete rather than abstract tasks (Zhang, 2002). Conscientiousness also has been studied as a "personality state" that changes over time, and has been found to be facilitated by task-oriented and externallymotivated situations (e.g., imposed, uninteresting, time-pressured tasks) (Fleeson, 2001(Fleeson, , 2007. State conscientiousness bears considerable similarity to what has been called the "strategic approach" to studying (Entwistle & McCune, 2004). Although active/metacognitive engagement may seem similar to conscientious engagement, the study strategies literature suggests strategic and deep approaches to learning are correlated but different. In fact, conscientious activities like "work discipline" have been found to be more related to achievement in a course (e.g., scores on knowledge tests) than deep processing strategies (Jansen & Bruinsma, 2005). Correspondingly, conscientious engagement may be more important than active engagement for ensuring participants achieve the goals identified and sought after by organizers of a public engagement event.

Uninterested Engagement
Practitioners may also find it useful to know if their methods resulted in boredom and disengagement, rather than intrinsically satisfying forms of engagement. Boredom reflects "the aversive experience of wanting, but being unable, to engage in stimulating and satisfying activity" (Fahlman, Mercer-Lynn, Flora, & Eastwood, 2013, p. 69). Not only is boredom associated with negative affect, difficulty concentrating, and slowed time perceptions, it is associated with low intrinsic motivation. This is important because intrinsically motivated persons tend to experience high task interest (Hidi, 2000), show greater acceptance of information and learning across situations (Keltner, Ellsworth, & Edwards, 1993;Pintrich, 2003), and demonstrate autonomous and active learning behaviors such as exploratory strategies and deep information processing (Hess, 2005;Keltner et al., 1993;Kirschner, Sweller, & Clark, 2006). Thus, interested engagement may provide motivational fuel for other forms of positive engagement. On the other hand, active learning and conscientious engagement could occur even when one is bored and uninterested, if, for example, one has extrinsic reasons to engage. Assessing uninterested engagement may be especially important to public engagement practitioners who are concerned, not just with learning, but also interest outcomes. The development of longer-lasting individual interest in topics (e.g., future engagement opportunities) often begins with temporary situational interest (Hidi & Renninger, 2006), which can be undermined by boredom.

Open and Creative versus Closed-minded Engagement
If, instead of increasing the public's knowledge or interest, a practitioner's goal is to maximize the amount of relevant information elicited from engagement participants (Rowe & Frewer, 2005), then the practitioner may find it useful to know if participants are engaging in a manner that encourages creative and divergent thinking and examination of the issue from multiple perspectives (Akbari Chermahini & Hommel, 2012;Nusbaum & Silivia, 2011). People differ in their trait open-mindedness, with those higher in openness to new ideas, art, emotions, activities, and values (Costa & McCrae, 1992;Goldberg, 1999) also tending to show more creative and divergent thinking (McCrae, 1987;Schilpzand, Herold, & Shalley, 2011). However, other predictors of states of creativity, openmindedness, and divergent thinking include positive affect (Akbari Chermahini & Hommel, 2012) and may include situations that combine safety and uncertainty (Haas & Cunningham, 2013). Deliberative practitioners may find it useful to know if such states are induced by their methods because individuals who engage in creative, divergent thinking tend to generate more original and appropriate ideas (Nijstad, De Dreu, Rietzschel, & Baas, 2010). Furthermore, if people are working in groups, it may be important that group members are open to the suggestions of others, so that information is shared and included in the groups' summaries rather than too quickly dismissed.

Angry Engagement
When topics of public engagement and deliberation include controversial and emotion-laden issues, the extent to which participants feel angry may also be relevant. Unlike openness and conscientiousness, anger tends to be more often examined as a situationally varying state than as a personality trait (but see Martin et al., 2000). Anger increases perceptions of human agency and blame (Keltner et al., 1993), which may undermine productive deliberation processes. In addition, because it tends to be a higher intensity emotion that is associated with "approach" behaviors (and feeling determined to reach some goal), it may result in the narrowing of cognitive scope-that is, it may reduce one's ability to attend to and take in information that is peripheral or unrelated to one's anger, and reduce ability to hold and manipulate multiple sources of information in working memory (C. Harmon-Jones, Schmeichel, Mennitt, & Harmon-Jones, 2011;E. Harmon-Jones, Gable, & Price, 2012). This suggests that angry or frustrated engagement may result in greater closed-mindedness, less learning, and may undermine active/metacognitive engagement. On the other hand, De Dreu, Baas, and Nijstad (2008) found anger associated with brainstorming more ideas of higher originality. In the De Drue et al. study, the source of angry mood and the creativity task were unrelated. The same effects may not be found if creativity and anger are assessed as responses to the same task.

Social Engagement
While deliberative public engagements typically include social interaction and discussion activities (Delli Carpini et al., 2004), it is possible for people to deliberate on issues alone. Social and collaborative learning theory suggests that interactions with others can help expose people to more ideas, make them more aware of their own knowledge gaps, and help them to see alternative perspectives on an issue (Dewiyanti, Brand-Gruwel, Jochems, & Broers, 2007;Johnson & Johnson, 1986). In addition, neutral and positive social interactions are positiveaffect inducing, which may further enhance creative and divergent thinking (Gokhale, 1995;Razon, Turner, Johnson, Arsal, & Tenenbaum, 2012). On the other hand, not all investigations find that group discussion or cooperative learning groups result in positive learning effects (Slavin, 1996) or better decisions (Surowiecki, 2005). Also, even within activities designed to be interactive, individuals and groups may vary in the extent to which they actually interact and share information with one another (Emich, 2012). Although many public engagement practitioners may presume that discussion and dialogue are keys to the success of the engagements (Delli Carpini et al., 2004;Rinner & Bird, 2009), only by examining the extent to which people indicate being socially engaged, will it be possible to begin to quantify the extent to which various effects depend on social engagement.

INITIAL SCALE DEVELOPMENT
Although the states of behavioral, affective, and cognitive engagement reviewed above may not reflect all possible forms of engagement, or even every important state of engagement, each is relevant to processes and outcomes likely to be of interest to practitioners and researchers of public engagement and deliberation. The Varieties of Individual Engagement (VIE) scales were developed through an iterative process that began with gleaning or deriving items from measures of study strategies (Martin et al., 2000;Shell & Husman, 2008;Shell et al., 1997), trait and state openness and conscientiousness (Fleeson, 2007;Goldberg, 1999), and existing scales that assess emotions and mood (Albrecht & Ewing, 1989;Watson, Clark, & Tellegen, 1988). Most of the scales required some revision to fit public engagement contexts. For example, measures of active/metacognitive engagement (e.g., Duncan & McKeachie, 2005;Pintrich & De Groot, 1990;Shell & Husman, 2008) are designed and specifically worded for formal educational (school) contexts, not public engagements.
Next, items were used in public engagements involving community residents and in deliberative engagements with students. Exploratory factor and reliability analyses conducted on earlier versions of the scales suggested that metacognitive/active learning engagement was correlated with but somewhat different from conscientious engagement, a finding we interpreted as consistent with previously mentioned findings that deep study strategies differ from strategic study strategies (Zhang, 2002). These early analyses also suggested that closedminded, open-minded, and creative engagement should be on different scales, despite their conceptual overlap with the construct of "openness." As part of our scale development, we also conducted cognitive interviews with students who had completed the items during classroom deliberative engagements, to explore their understanding and interpretation of the items. These interviews helped us to identify items that students had difficulty understanding (e.g., many were unfamiliar with the adjective "conscientious"), that resulted in varied interpretations, or that were less relevant to public engagement situations (e.g., although we thought some engagements might evoke "competitiveness," a number of participants felt competitiveness was irrelevant). After this initial development, the VIE scale items were reduced and revised to assess five positive engagement scales: conscientious, active learning, open-minded, creative and social engagement; and three negative engagement scales: uninterested, angry and closed-minded engagement.

THE CURRENT STUDIES
The present studies first examine the structural validity of the VIE scales using confirmatory factor analyses. To test whether the measures assessed the same latent constructs across different tasks and samples (De Ayala, 2009), we replicated all analyses in two tasks using one sample of students (Study 1, Time 1 and Time 2), and then in a new adult sample involved in a community participatory budgeting activity (Study 2). Our primary hypotheses were that items for each scale would form a unidimensional and internally consistent scale, and the items would show a similar structure across the tasks and samples. We also expected that certain scales (e.g., the cognitive scales or the positive or negative scales) might intercorrelate and create higher order factors that would be evidenced across tasks and samples. Thus, we conducted analyses and comparisons of alternative structural models. Second, after examining the structural and internal consistency of the scales, we examine evidence for their convergent, divergent and construct validity by examining their correlations with one another and other elements drawn from Figure 1 (e.g., personality traits, task perceptions, and knowledge gains).

Participants and Procedures
Study 1. Participants were 349 undergraduates enrolled in an introductory biology course at the University of Nebraska in the fall of 2011, including 153 (43.8%) males and 194 (55.6%) females (2 unreported). The mean age was 19.4 years old (SD = 2.32), with the majority identifying as freshman (37.9%) or sophomores (37.6%). 4 Over the course of the semester, and as part of a larger study of public engagement, the students completed four assignments concerning ethical, legal and social issues (ELSI) related to nanotechnology. 5 Demographic and personality measures were administered as part of the first assignment in the series.
Knowledge measures were administered during the first and last assignments (assignments 1 and 4). To minimize retrospective biases, the VIE scales were administered immediately following tasks that took place during the other two assignments (assignments 2 and 3). Specifically, the VIE scales were administered immediately after a reading assignment about biological applications of nanotechnology (Task 1), and immediately after an in-class deliberation about ethical scenarios involving nanobiological technologies (Task 2).
We intentionally varied student activities during these tasks. Task 1 reflected a 2 x 2 experimental design. The first experimental manipulation randomly varied the organization of the information so that it was either in expository paragraph form (a form people commonly encounter when reading news articles), or organized more explicitly in terms of "pro" and "con" perspectives (a presentation format commonly encountered in deliberations). The second experimental manipulation randomly varied whether students received or did not receive critical thinking prompts designed to enhance active/metacognitive and conscientious engagement. Task 2 varied the social context. Students were randomly assigned to write their responses to the ethical scenarios either in the context of a moderated small group discussion or while working on the same tasks individually in a quiet room. The advantage of including data from across the varied cognitive and social conditions is that the differing conditions should create variation in the engagement items, allowing us to test whether the items varied together as expected.
Study 2. Participants were 450 community residents, including 237 (53%) males and 204 (45%) females (9 unreported), who were participating in an online participatory budgeting activity and survey during the summer of 2012. 6 The average age of participants was 50.2 years old (SD = 14.1). The majority (95%) were white, and had college or higher levels of education (29% had a bachelor's degree, 12% had some graduate school, and 28% had advanced degrees).
Most of the VIE items were administered near the end of the online survey, which primarily focused on a participatory budgeting activity. The budget activity asked participants to read about nine city programs and decide which of the programs should be funded or cut during the next fiscal year. Upon making their choices, participants received automatically generated, individualized feedback regarding the impacts of their choices on the city budget and property taxes and then could change their program choices. After completing their budget decisions, participants reported their satisfaction with the task, their subjective knowledge about city budgeting topics, and their responses to 13 randomly chosen VIE items. 7 After they finished these items, participants were given the option to "answer additional questions for research purposes," including the rest of VIE items, measures of dispositional trust and trust in government, and additional task perceptions questions.

Measures 8
Varieties of Individual Engagement. In both studies, the items listed in the appendix were used to measure eight engagement states: conscientious, active learning, open-minded, social, creative, uninterested, angry, and closed-minded engagement. Each item was preceded with the stem "during the (assignment, Study 1; budgeting activities, Study 2), I…" and was designed to reflect one of eight hypothesized forms of engagement (e.g., "felt focused" for conscientious engagement). Participants rated each item on a 5-point scale where 1 = Not at all and 5 = A great deal.
Knowledge Measures. In both studies, we included measures of knowledge. In Study 1, we assessed both objective and subjective knowledge. Objective knowledge was measured as the total correct of five true/false questions relevant to the background reading. Subjective knowledge was assessed with three items asking participants to rate their familiarity with the topic (e.g., "How familiar are you with nanotechnology" and "…with how nanotechnology is used in medical research and development") using a 5-point scale that ranged from 1 = not at all familiar to 5= extremely familiar (α = .87). In Study 2 we assessed subjective learning using 3 items asking about perception of knowledge gains (e.g., "I learned a lot about the budgeting process as I completed the budgeting task," α = .82). Participants responded to each item by choosing a point on a 7point scale ranging from 1 = strongly disagree to 7 = strongly agree.
Personality Measures. In Study 1, participants also completed personality measures of trait openness, dispositional trust, and need for cognition. Trait openness was assessed with the eight items from the International Personality Item Pool (α = .78) (see IPIP.ori.org) corresponding to "openness to experience" in the Big-Five or Five Factor Model of personality (Costa & McCrae, 1992;John & Srivastava, 1999). Dispositional trust, which refers to the extent of individual's expectancy that other people can be relied upon (Goto, 1996;Rotter, 1967) was assessed with five items from the IPIP (α = .79) (see IPIP.ori.org). Need for cognition refers to tendencies to exert cognitive effort and enjoy cognitively demanding tasks, and was assessed using seven items from the short version of the Need for Cognition Scale (α = .79) (Cacioppo et al., 1996). Participants responded to the items assessing these three personality constructs using a 6-point scale ranging from 1= strongly disagree to 6 = strongly agree.
In Study 2, participants completed two measures of trust. Dispositional trust was assessed with the three bipolar items from the General Social Survey (GSS) (James & Smith, 1992). These items use a 1-10 point response scale with labels at each end (e.g., 1 = "Most people can be trusted" to 10 = "You can't be too careful in dealing with people," α = .83). However, items were recoded so that high scores would indicate more trust. Trust in city government was assessed with three items asking participants to rate their confidence in "city government," "city council" and "the Mayor's office" on a 5-point scale ranging from 1 = no confidence to 5 = total confidence (α = .90).
Task Perceptions. In Study 2, participants also answered questions about their perceptions of the engagement material and tasks, including three questions about their perceptions of the quality of information provided as part of the budgeting task (e.g., "The background information about programs was not very helpful," reverse coded, α = .74), and a six-item measure of their perception of autonomy support during task-that is their feeling about the extent to which the task gave them choice and control, perceptions which are fundamental to intrinsic motivation ) (e.g., "The budgeting task was structured so that I could choose for myself what information was most important to my decisions," α = .78). Participants answered using a 7-point scale ranging from 1 = strongly disagree to 7 = strongly agree.

Data Screening and Preliminary Analyses
Missing Data. In Study 1, the software administering the engagement measure reminded or required students to reply to all items, resulting in little missing data. However, not all students involved in the course agreed to have their data analyzed, or completed all engagement activities. Of those enrolled in the course, 88% gave consent to have their data used in the present study. After omitting non-consents or absentees, there were data from a total of 320 students for Task 1 and 313 for Task 2.
In Study 2, a total of 531 persons completed at least some of the additional "optional" measures at the end of the survey, which included the items from the VIE scales. Cases with over 50% missing values on the VIE items were deleted, resulting in 450 valid participants. 9 Originally 40 VIE items were administered; however, two items were identified for which over 50% of participants had missing values. The large amount of missing data on these two items may indicate they were viewed as inappropriate or irrelevant by the participants (both also poorly fit their hypothesized factors in Study 1; see footnote 12). Thus, these two items were dropped from further analysis. With these items removed, the majority of the participants (81%) had no missing data on the remaining 38 items. Those who had missing data only omitted less than 15% of the VIE items. Pairwise deletion was used to address the remaining missing cases for all CFA analyses. Data Screening. Item-level data were screened for outliers and the skew and kurtosis values were examined for non-normality. No item-level outliers were detected but the majority of the engagement items were markedly nonnormal with skewness or kurtosis values larger than 2.3 (Lei & Lomax, 2005).
Preliminary, Single-Factor Analyses. To examine the unidimensionality and internal reliability of each of the scales, we conducted CFA analyses using MLR estimation on each of the hypothesized factors individually. 10 Despite the non-normality of many items, only the items from the angry scale showed severe piling up of the data at one end of the scale. 11 Raykov and Marcoulides (2011) suggest that when items have at least five response options and responses do not pile up at one end of the scale, omega (a CFA model-based measure of reliability that is interpreted similar to Cronbach's alpha) can be computed based on values from robust maximum likelihood estimation (MLR). These preliminary analyses, which allowed us to compute omega and identify sources of local misfit, resulted in our dropping of two items due to low loadings and/or large residual covariances, results suggesting that the items were redundant and that shared variance beyond the factors should be measured. 12 As shown in Table 1, good internal reliability of each final individual scale was indicated by the omega values (which ranged from .73 to .87). For comparison, Cronbach alphas are also listed and also showed adequate to good reliability, with values ranging from .62 to .94. Note. S1 = Study 1, S2 = Study 2, T1 = Task 1, T2 = Task 2. *The omega values for the Angry scale may not be accurate due to its extreme non-normality in these samples (see footnote 11).

Full Eight-Factor Model
Next we examined an eight-factor model in which each item loaded only on its designated factor and the factors were allowed to freely correlate. Given the marked non-normality of some items and consistent with recommendations by Bentler and Chou (1987), we examined our full eight-factor model using methods appropriate for ordinal data. Furthermore, because the sample sizes of our studies were less than 1000, the confirmatory factor analysis were run with a robust weighted least square estimator WLSMV, which is more appropriate for handling ordinal data with smaller samples (Flora & Curran, 2004). Final model fit statistics include Mplus's WLSMV variance-adjusted chi-square, its degrees of freedom, and its associated p-value; the Comparative Fit Index (CFI); the Root Mean Square Error of Approximation (RMSEA); and Weighted Root Mean Square Residual (WRMR). Good model fit is suggested when the chi-square is non-significant, CFI > .95, RMSEA < .06, and WRMR < .90 (Hu & Bentler, 1999;Yu, 2002).
As shown in Table 2 (top one third of table), across all three analyses, the chi-square tests were significant, indicating that the data did differ significantly from the hypothesized model. However, chi-square tests are known for being overly sensitive to misfit, leading many to recommend consideration of alternative fit indices (Gerbing & Anderson, 1993). Across all three analyses, the eight-factor model showed good fit by two indices, RMSEA (always < .05) and CFI (always > .95); however, WRMR was consistently greater than the criterion desired for good model fit (always > .90). Although the failure of the models to meet the WRMR criterion could indicate further development is needed, WRMR is an experimental index and it has been recommended that it may be ignored if other indices show good fit. 13 With regard to individual items, as shown in the appendix, the item factor loadings from each of the three CFA analyses ranged from .475 to .971, indicating acceptable to good local fit of each engagement item to its factor in all three analyses. Note. All Chi-square values are significant as noted in the text, ***p < .001, * p < .05.

Inter-Factor Correlations
We next examined the inter-correlations between the factors (which had been allowed to freely correlate in our CFA models). Table 3 shows both the CFA factor correlations (from the CFA models, and based on shared item variance), and the correlations between scales when computed as averages across items in the scale (as practitioners may use the scales). As shown in the outlined rectangles in Table 3, all the positive engagement factors were positively correlated and the negative engagement factors were positively correlated with each other. The positive engagement factors were either negatively or not significantly correlated with the negative factors, with the exception of social engagement. In Study 1 Task 1, social engagement had a small but significant positive correlation with closed-minded engagement, which we had not hypothesized.  Other notable correlations in all three data sets include relatively high correlations between conscientious, open-minded, and active/metacognitive engagement (most CFA rs > .70), and between creative and active/metacognitive engagement (again, CFA rs > .70). Open-minded and conscientious engagement were more highly related than expected, suggesting the possibility that people feel they "should" be open-minded, or that being open-minded is part of what it means to do public engagement tasks conscientiously. Among the negative traits, angry and uninterested engagement were rather highly correlated as well (CFA rs > .6).
On the other hand, creative, open-minded, and closed-minded engagement were not all highly related, despite personality theories suggesting they might be. Open-minded engagement did relate moderately with closed-minded and creative engagement; however, creative and closed-minded engagement were not significantly related in any of the data sets. Furthermore, creative engagement was more highly related to active learning engagement, than to open-minded engagement. These results suggest that, even if creative, open-minded, and closed-minded constructs are related at the level of personal dispositions, it may be useful to examine them separately within specific situations. We also found that angry engagement was usually unrelated to creative engagement, suggesting that people don't perceive themselves as either more or less creative when frustrated or angry. This result is consistent with lack of a relationship between creative and closed-minded engagement, but is somewhat inconsistent with prior research suggesting that angry moods can relate to more originality of ideas (De Dreu et al., 2008).
Although most of the relationships were consistent across the three data sets, a notable difference was in the correlations between social engagement and negative engagement states. Social engagement was positively correlated with angry engagement in Study 2 but negatively in Study 1 Task 2. Social engagement was negatively correlated with feeling uninterested for both Study 1 tasks, but unrelated to uninterested in Study 2. Finally, while social usually was unrelated to closed-minded engagement, it was positively related in Study 1 Task 1. It is possible that different correlations with social engagement emerged across samples because social engagement may have different functions in different contexts. For example, Study 1 Task 1 was comprised of individual homework, not group work, and students would have had to self-seek (perhaps like-minded) others to socially interact. This may have facilitated closed-minded engagement. Meanwhile, for half of the students, Study 1 Task 2 took place with randomly assigned classmates who may have had differing views that, when shared, reduced closed-mindedness and angry engagement. Alternatively, when public engagement activities neither restrict nor provide a platform for social interactions (e.g., in Study 2), participants who feel angry may be more likely to actively seek social interactions.

Exploring Other Factor Structures
Because the positive engagement factors of conscientious, open-minded, active learning and creative engagement were highly intercorrelated, and because they also are theoretically related (e.g., open-mindedness and creativity are related in personality theories), we next tested two additional models. First, we tested a model in which the uninterested, closed-minded, angry and social factors were kept the same (as separate factors), but the active learning, conscientious, open-minded, and creative items were assigned to load on a single, first-order "positive engagement" factor. The fit statistics for this new five-factor model were poorer than for the eight-factor model. As can be seen in Table 2, in most cases the model did not meet the cutoff criteria for acceptable fit. To further explore the reduction in fit, we used the DIFFTEST option within Mplus to obtain the Chisquare difference test to account for the variance adjusted Chi-square that is utilized in WLSMV estimation. In all three datasets, the fit of the eight-factor model was better than the five-factor model (Study 1 Time 1: χ 2 (18) = 235.52, p < .001, Study 1 Time 2: χ 2 (18) = 209.94, p < .001, Study 2: χ 2 (18) = 302.99, p < .001). Second, we examined a second-order factor model in which the positive engagement items first loaded on their own subfactors, and then comprised a higher order "positive engagement" factor (see Figure 2). As can be seen in Table  2, the fit statistics for this model indicated somewhat less fit than the eight-factor model, but in general levels were acceptable. As the higher order factor is not nested within the eight-or the five-factor models, chi-square difference tests cannot be used to statistically compare the second-order factor model with the other models. Nonetheless, overall, the evidence indicates that treating the eight scales separately is ideal, either as an eight-factor model or with the positive engagement items entered into a higher order factor structure.

Construct Validity
Beyond measurement validity, the ultimate utility of the VIE scales will depend on whether they assess conceptually and theoretically useful constructs that connect the constructs described in Figure 1. To begin to assess the construct validity of the scales, we examined relationships between each of the scales and theoretically relevant public engagement features ( Fig. 1 element A), personality traits (element B), perceptions of the engagement tasks (element C), as well as post-deliberation measures of subjective and objective knowledge (element E). Because practitioners are most likely to use the scales by averaging across items rather than computing a CFA model, for these analyses, all engagement scale scores were computed using means across items for the scale.

A. Public Engagement Features.
To examine the scales' sensitivity to different public engagement design features, we compared the engagement scores obtained under the different cognitive and social conditions randomly assigned during Task 1 and 2 of Study 1. Specifically, we conducted a series of 2 x 2 analyses of variance 14 to examine the main and interactive effects of (1) the different background organizations (paragraphs vs. pro/con lists) for which we had no specific engagement-related hypotheses, and (2) use or non-use of the critical thinking prompts designed to promote active/metacognitive engagement, on each form of engagement assessed during Task 1. The information organization condition had no main effects on any of the engagement states and also did not change the pattern of cognitive effects. 15 Thus, we only report the cognitive comparisons (critical thinking vs. control group) for each engagement state in Table 4.
As shown in Table 4, the critical thinking versus control experimental condition comparisons revealed significant differences on most of the engagement states, except creative and angry engagement. Although the effect sizes were small, when significant differences were found, students in the critical thinking 14 We also attempted a multivariate analysis. However, because the Box's Tests of homogeneity of variances for both Task 1 and 2 were significant, we conducted analyses for each engagement state separately. 15 There was only one significant interaction discovered in the 2x2 analyses. The pattern of this interaction was such that the impact of the cognitive condition on states of conscientious engagement was somewhat stronger (but in the same direction) when the background information was organized in a pro/con format than when it was organized in paragraphs. condition had significantly higher scores on the positive engagement factors, except social engagement, upon which they scored significantly lower. In addition, students in critical thinking condition had significantly lower scores on the closed-minded engagement scale, compared to students in the control condition, but also reported greater uninterested engagement. The largest effect sizes (though still small) were for the conscientious and active learning engagement measures-which were the forms of engagement we had predicted might be most likely to be impacted. Taken together, the results suggest that the critical thinking prompts inspired many forms of positive engagement (e.g., active learning and conscientiousness, open-mindedness and reduced closedmindedness), but also undermined social engagement and increased boredom during the reading homework assignment. Notes. *p < .05, **p < .01. N = 315 and degrees of freedom for all analyses are (1, 313). ω 2 is the effect size, the total variance of dependent variable accounted by group variable other than within-group variance; .01 is small, .06 is medium, .14 is large (Cohen, 1988).
We also examined whether levels of engagement differed between those students working alone or in groups during Task 2, when responding to the ethical, legal, and social scenarios. As shown in Table 5, the largest significant difference was found for social engagement, with those in group conditions reporting more social engagement than those in alone conditions, as expected. Most other engagement states also showed significant differences between conditions, except angry and closed-minded engagement. Specifically, students in group condition showed significantly higher scores on all the positive engagement factors, and scored significantly lower on the uninterested engagement scale. The effect sizes ω 2 were small to large, ranging from .02 to .64. In general, the effect sizes were larger for the Task 2 than the Task 1 comparisons. Thus, in our study, the variation in social aspects of public engagement design had a greater impact on engagement than variations in the cognitive tasks we examined. Note. *p < .05, **p < .01, N = 271, and degrees of freedom for all analyses are (1, 269). Table 6, at both time points in Study 1 (Task 1 and Task 2), the engagement factors showed highly similar patterns of correlations with personality variables. Trait openness, which should generally predict open-mindedness to new topics (e.g., nanotechnology), positively correlated with most of the positive engagement factors and negatively correlated with uninterested engagement during both tasks. Trait openness also had some of its highest positive correlations with open-minded and creative engagement. Interestingly, trait openness was not significantly negatively correlated with closed-minded engagement at either time point, supporting the idea that closedminded engagement, at least as measured by the VIE scale, might be different from low open-minded engagement, and that the two should be examined separately.

B. Personality Traits. As shown in
Need for cognition, which assesses tendencies to exert cognitive effort and to enjoy cognitively demanding tasks, was positively correlated with most of the positive engagement factors, especially with active learning and conscientious engagement, as expected. It was also significantly and positively correlated with creative engagement. In fact, during Study 1 Task 2 (responses to the hypothetical ethical scenarios), need for cognition was most highly correlated with creative engagement. In addition, it was negatively correlated with uninterested engagement during both Study 1 tasks. It seems logical that persons high in need for cognition would find the reading and deliberating tasks used in this study more interesting than those low in need for cognition.
In Study 1, dispositional trust, which indicates a general tendency to trust others, was negatively correlated with angry and closed-minded engagement during both tasks, but was unrelated to the positive types of engagement. The negative correlations make theoretical sense because anger involves perceptions of blame that might be more likely when one distrusts others, and closedmindedness might prevent one from entertaining untrustworthy information. On the other hand, distrust could enhance perceptions of intentional deception and influence anger. Distrust could also make people defensive and risk-averse, resulting in closed-mindedness. Again, the lack of a relationship between trust and open-minded engagement suggests that open-and closed-minded engagement are somewhat different things. Unlike in Study 1, dispositional trust in Study 2 was not correlated with any of the engagement factors. It is possible that different results were obtained because of the different measure of dispositional trust. In Study 1, we used items from the personality literature, but in Study 2, we used items from the General Social Survey (GSS) (James & Smith, 1992). Alternatively, persons in Study 2 were likely more familiar with city government than students were with those organizing their engagement activities. This may have resulted in the citizens' engagement being less affected by dispositional trust, and more affected by specific assessments of trust in city government (Hamm et al., 2013). In support of this idea, trust in city government positively correlated with open-minded engagement, but negatively with uninterested and angry engagement. Thus, the only consistent trust-engagement correlation is with (reduced) angry engagement.
C. Individual Perceptions. Self-determination theory  proposes that environments that support people's autonomy (e.g., that provide choices and personal control instead of exerting pressure through external punishments and rewards) promote intrinsic motivation and greater task engagement. Consistent with this theory, our results from Study 2 (see Table 7) indicated that perceptions of autonomy support significantly correlated with all the engagement factors in expected directions, with the exception that social engagement was negatively correlated with autonomy (lending further support to the idea that social engagement may be positively or negatively motivated). Perceived autonomy was especially highly correlated with less angry engagement. Perceived quality of background information also correlated with most of the engagement factors. Its correlations with the engagement factors were similar to those from perceived autonomy, but it did not correlate as highly with angry engagement, and correlated more highly with open-minded and closed-minded engagement.  Table 8, the correlations involving objective knowledge were quite small, which might be expected given that this knowledge measure was comprised of only five true/false questions. Nonetheless, some significant correlations did emerge. During Study 1 Task 1 (reading about nanotechnology), creative engagement was significantly (and negatively) correlated with final measures of objective knowledge. This suggests that engagement in creative thinking while reading (which may have been inspired because participants were asked to try to think of potential ethical, legal, and social issues related to nanotechnology as they read), may undermine attention to some of the factual information presented in the background document. Meanwhile, during Task 2 (deliberation about the ethical scenarios) conscientious engagement positively predicted, and closed-minded and angry engagement negatively predicted, post-measures of objective knowledge. It is possible that conscientious engagement during consideration of the ethics scenarios (Task 2) was more necessary for making connections back to the original factual material, than it was when students were focused more directly on that factual material (i.e., during Task 1 reading). Alternatively, because the students answered the knowledge questions during the week right after Task 2, temporal proximity may have allowed for greater relationships between Task 2 engagement and the knowledge measure. Notes. * Significant at the p < .05 level, ** Significant at the p < .01 level. S1 = Study 1. Read = Reading task (Task 1). Delib = Deliberation task (Task 2). Study 1 Ns = 263-315; Study 2 Ns = 429-449.

D. Knowledge Outcomes. As shown in
Subjective knowledge (i.e., Study 1 students' self-assessed familiarity with the topics at the end of the activities) and subjective learning (Study 2) were consistently positively correlated with each of the positive engagement factors, with the exception of social engagement (see Table 8). It also negatively correlated with uninterested (during both tasks) and angry engagement (during the reading task), but not with closed-minded engagement. Overall, the higher correlations between subjective knowledge and engagement variables (compared to those involving objective knowledge) may be partly because both are subjective self-report measures. On the other hand, the small but sometimes significant correlations between some of the engagement factors and objective knowledge suggest that the engagement factors are measuring more than just selfreports and may have utility predicting knowledge outcomes.

GENERAL DISCUSSION
The primary purposes of this article include (a) proposing a set of measures of potentially important behavioral and psychological states that may be elicited during public engagements, and which may impact the outcomes of public engagements, (b) examining the structural and psychometric validity of the scales, and (c) assessing evidence for the convergent, divergent and construct validity of the scales, as well as their potential usefulness in public engagement contexts. The eight different "varieties of individual engagement" constructs we examined were drawn from prior research in personality, social, and educational psychology. Such constructs included psychological states that have been found in other fields to relate to some of the outcomes sought after by practitioners of public engagement (e.g., learning). The scales themselves were often adapted from prior scales, in order to be able to be used across public engagement contexts, ranging from those designed to present information to the public, to those designed to gather information from or to dialogue with the public (Rowe & Frewer, 2005).

Evidence Supporting the Scales
In both Studies 1 and 2, confirmatory factor analyses supported the structure of the scales. The final items comprised an eight-factor structure, strongly supported by all three of our CFA analyses. These studies involved diverse engagement activities ranging from reading about topics, small group discussions, and completion of deliberative online activities, thus providing initial evidence that the scale structures will be valid across varied public engagement contexts. Furthermore, the engagement scales comprised of these items showed adequate to excellent internal consistency across both studies.
The VIE scales not only showed hypothesized relationships with each other, but also correlated as expected with other constructs, such as trait openness, need for cognition, dispositional trust, subjective knowledge, autonomy, and participant task perceptions. For example, across both studies, trait openness showed similar patterns of relationships with open-minded engagement. In addition, the different experimental manipulations used in Study 1 demonstrated expected significant effects on the engagement states. Active learning and conscientious engagement were especially enhanced by the randomly assigned critical thinking prompts in Task 1, and social engagement was strongly and significantly higher among participants in the social condition during Task 2, than among those in the individual condition.

Future Scale Development
There are certain limitations to the scales that suggest that they would benefit from further development. First, as previously noted, the scales may not address all possible varieties of engagement, and there are several others that might be useful. For example, if you compare our dimensions to those of other emotion or political science researchers (e.g., MacKuen, Wolak, et al., 2010), you will see that anger is but one form of aversion that is assessed, and we did not assess enthusiasm or anxiety. Future research might examine these additional varieties of engagement states, as well as others, such as states of empathy or disgust, which could impact participant interactions and information processing (e.g., Lamm, Batson, & Decety, 2007;Rozin, Markwith, & Stoess, 1997).
Second, not all of the model fit indexes showed good fit of the model, although the existing results were enough to indicate an acceptable model, and the one index indicating poor fit (WRMR) is an experimental statistic in need of further study. In addition, open-minded and closed-minded engagement had somewhat lower internal reliabilities within the adult sample of city residents. These scales in particular could benefit from further refinement and testing in adult samples. In a related vein, although we imagine that the scales may be used individually, these items were tested as a set and additional studies should be conducted to examine how individual scales behave when administered alone or as shorter scales. Development of short versions of the scales would be especially useful in case practitioners are interested in measuring all of the engagement states at one time. In the appendix, we identified the items we suspect likely to comprise the best three-item short version of each of the scales, based on the CFA analyses, and our future work will test these short scales.
Some questions also remain about the relationships between the scales. For example, states of open-minded and conscientious engagement were very highly related in both studies. This was not hypothesized because, within the personality and social psychology literature, conscientiousness and openness are seen as distinct and relatively independent constructs. It could be that high correlations occur in public engagement contexts because people who are feeling conscientious (which is typically conceptualized as having a normative or moral component to it) feel they "should" be open-minded when they are in public engagement contexts. The extent to which these two scales truly measure distinct underlying forms of engagement might be explored by devising public engagement contexts that suggest different norms (e.g., engaging people with others who encourage and demonstrate closed-mindedness).
Finally, although the engagement states were examined close in time to the actual engagement experiences (immediately after the activities), the data were still retrospective self-reports, and thus subject to the typical weaknesses of such measures (Howard, Millham, Slaten, & O'Donnell, 1981;Metts, Sprecher, Cupach, Montgomery, & Duck, 1991;Scollon, Kim-Prieto, & Diener, 2003). The scales nonetheless were both predictive of other self-reports (e.g., participant personality traits, task perceptions, and subjective knowledge), as well as measurably affected by our experimental manipulations (the critical thinking prompts and social situations). On the other hand, the scales were only weak predictors of objective knowledge. It is possible that our objective knowledge measures (comprised of only a small number of questions), were not ideal measures, and with further refinement of both the knowledge and the VIE scales it may be possible to improve prediction of such outcomes. Further development of the VIE scales also should focus on their ability to predict other outcomes of importance to engagement researchers and practitioners, such as development of deliberative values and civic engagement.

Researcher and Practitioner Use of the VIE Scales
The VIE scales constitute a promising tool for beginning to focus, within public engagements, on potentially important mediators that may then lead to targeted outcomes. Although the VIE scales may benefit from future development and refinement, the face validity of the scales and the evidence supporting their psychometric validity suggests that they could be useful to practitioners as well as researchers. Researchers might use the scales to begin to examine hypothesized impacts of deliberative discussions, and features of deliberative engagements that are theorized as important active positive ingredients. The scales may be particularly useful to begin to examine the joint effects of personality and design features of deliberative activities. For example, it is well known that people vary in their preferences for difficult cognitive tasks (Cacioppo et al., 1996), which deliberative activities may often resemble. Thus, a representative sample of deliberants is likely to include persons high and low in need for cognition. The present scales could be used to advance theory and research regarding potentially different means of maximally engaging such different groups.
Our work also provides an example of how practitioners may use the scales, iteratively, to improve their methods. During development studies that took place prior to Study 1, we used previous versions of the VIE scales to explore the effectiveness of our developing critical thinking supports, only to find that students generally disengaged rather than engaged in response to early versions of those prompts . By reflecting on the VIE results and student input, we were able to make revisions to the critical thinking activities to improve their impacts, as shown in the results from Study 1 of this article, which suggested that the critical thinking prompts successfully elicited a number of positive forms of engagement, such as active learning, conscientious, and open-minded engagement (but also increased boredom and disinterest). Thus, the VIE scales can allow practitioners to examine a profile of the impacts of their engagement activities, and help them reflect in a more nuanced way concerning their desired versus actual impacts on engagement participants. As another example, in the case of public engagements involving controversial issues, practitioners may wish to reduce participants' negative engagement states, such as angry or closed-minded engagement. On the other hand, positive engagement factors like conscientious and active learning engagement may be applicable in most engagement situations because they are crucial to getting insightful information from the public.

Conclusions
The empirical results in this article support the proposed structure and reliable assessment of eight varieties of individual engagement. Establishing this measurement validity is an important first step for researchers interested in testing whether different experiences of engagement mediate the relationships between engagement design factors, participant individual differences, and desired outcomes. It is also important for practitioners who may wish to use the scales to assess the impacts of their design choices. Correlations with situational and individual characteristics such as personality variables, task perceptions, and important outcomes also suggest their viability as mediators, making it possible to next test more specific models within our theoretical framework (illustrated in Figure 1), or simply assess engagement quality. Through the use of such scales, practitioners and researchers will begin to be able to compare results across studies, and to make progress understanding what public engagement features work for what purposes and why . Furthermore, because the VIE scales are grounded in the established psychological literature, the use of these scales may facilitate connections with such literatures, greater application of psychological theories to the public engagement context, as well as allowing public engagement research to contribute to these literatures. Note: Model 1 N = 320 ; Model 2 N = 313 ; Model 3 N = 450. Response scale ranged from 1 = not at all to 5 = a great deal. *Items nominated for short form 3-item scale.