Developing an Innovation Attitude Survey for Middle School Students

This article describes the development and validation of an Innovation Attitude Survey (IAS) composed of 16 Likert-type items selected to measure middle school students' attitudes toward innovation and leadership in the advancement of new ideas. The goal of developing the IAS was to identify desirable dispositions that may be related to future STEM careers. Data gathered from 764 middle school students from one state in the United States in 2021 were used to validate the instrument and establish the measurement properties of the instrument's scales. Factor analysis revealed three stable constructs representing a) propensity to be inventive, b) motivation or pride, and c) leadership, which were confirmed with multidimensional scaling techniques. Internal consistency reliability was found to be excellent for the total survey, very good for two of the separate scales, and acceptable for the third scale. Since the IAS was created to provide measures for a project-based STEM education program, educators may find this instrument useful for assessing pre-to-post intervention changes as well as for identifying differences in selected groups of students. Tangential findings from this instrument validation exercise indicate that the IAS can add valuable knowledge to STEM education in areas not previously assessed at the middle school level but potentially important for the pursuit of STEM careers.


Introduction
Politicians, educators, and business leaders have recognized that creativity and innovation are central to economic success and are needed to solve pressing social problems (Sawyer, 2012).Preparing students for future careers involves paying attention to skillsets employers are seeking.
A summary of multiple research reports focused on skills needed for the future found innovation skills to be an important category that includes creativity, critical thinking, collaboration, and communication (Eich, 2021).Teaching the skills needed to develop innovative thinking is critical for making individuals more employable and competitive (Bacigalupo et al., 2016).While many universities have been adding courses that include innovation, such courses may be too late to alter long-persisting attitudes (dispositions) developed during pre-college schooling in which the curriculum was focused on standardized testing rather than creativity and innovation (Wright & Jones, 2016).Attitudes formed during middle school influence students' academic performance (Liu et al., 2011).Middle school is an appropriate age to develop an interest in areas such as science that will persist through secondary school, into college, and beyond into a career (Christensen & Knezek, 2018; Baran et al., 2019).Impacting middle school students' interests and intents for future careers is critical as they prepare to take the courses in high school that may set trajectories for a career (Misiti et al., 1991).Improving the science, technology, engineering, and mathematics (STEM) workforce continues to be a top priority for policymakers, practitioners, and researchers who need to recruit and retain more students to work in STEM-related fields (Heilbronner, 2011).
Preparing students for future careers involves a focus on 21 st century skills defined by many different organizations (Eich, 2021) but having as commonalities critical thinking, creativity (innovation and invention), collaboration, and communication (Partnership for 21 st Century, 2011).The World Economic Forum (2020) Future of Jobs Report identified the top 10 skills that 50% of people would need to attain within the next five years to compete in the transforming job market.Analytical thinking and innovation are at the top of the required skills list, followed closely by critical thinking, leadership, resilience, and problem solving (World Economic Forum, 2020).
Innovation is critical to economic growth (Geotab Team, 2021).Fostering the next generation of STEM innovators has the potential to change the world by solving complex world problems (Geotab Team, 2021) and improving job growth, employment rates, and wages (Snyder, 2019).Addressing skills such as innovation in schools has not traditionally been considered part of the school curriculum.However, more recently, educators have recognized the need to include skills often referred to as "soft" or noncognitive skills.Because of the importance of noncognitive skills in future careers, developing these skills should be an explicit goal of education (Garcia, 2014).Identifying and quantifying these middle school students' perceptions of and attitudes about these concepts and skills can help school leaders strengthen their curricular programs and assist students in developing these important competencies.
This article describes the development and validation of the Innovation Attitude Survey (IAS), an instrument designed to measure middle school students' attitudes toward innovation, STEM dispositions, STEM career interests, and leadership in advancing new ideas.The authors' efforts were part of a state-wide, STEM-focused curricular enhancement program that employed project-based instruction.The development of this instrument was guided by self-efficacy theory and several attributes deemed important for measuring innovation, such as a propensity for being inventive and persevering to the completion of innovative ideas.

Theoretical Perspectives and Literature Review
Self-efficacy can be defined as confidence in one's competence and is known to influence the motivation and confidence in engaging in an activity (Bandura, 1977).Self-efficacy is a necessary condition for creative productivity and discovering new knowledge (Bandura, 1997).The IAS is rooted in the concept of self-efficacy as it relates to innovation-related attributes.Researchers have used self-efficacy as an evaluation component for entrepreneurial education, finding a strong correlation between entrepreneurial self-efficacy (ESE) and creativity as well as a strong correlation between ESE and leadership (Barakat et al., 2014).Other researchers have also used self-efficacy as a construct to understand creativity in the workplace and concluded that creative self-efficacy could predict creative performance (Tierney & Farmer, 2002).According to Bandura (1997), self-efficacy is a good predictor of behavior.
Innovation has been described by Isaksen et al. (2010) as having a close and complementary relationship with creativity in problem-solving.Scholars have pointed out that creative ideas are often the foundation for innovation (Scott & Bruce, 1994;Wright, 2012).While creativity and innovation are closely related, creative thinking differs in that it may include a wide range of ideas that are not useful, whereas true innovation must be both unique and useful (Price, 2019;Runco & Jaeger, 2012).
There are other attributes beyond creativity that are necessary for the development, implementation, and completion of ideas.Two attributes that also contribute to the success of ideas are motivation and leadership.Motivation to act on one's beliefs is important in enacting change (Sinatra et al., 2012).Prior research has reported motivation as the single most important factor in whether a student learns (Pintrich & Schunk, 1996;Stipek, 2002).Motivated students set higher goals for themselves and persist in learning (Koca, 2016).Tangential to motivation is pride in what one is doing.Tracy (2016) views pride as intrinsic motivation, a major reason people manage to accomplish difficult things.
Leadership has been defined as any behavior that influences the actions and attitudes of others to achieve certain results (Wade, 2017).Leadership is often conceived of as a process where one or more persons influence a group to move in a certain direction (Ibrahim & Daniel, 2019).Leadership is sometimes described as being "in influence" vs. "in control" (Bowman, 2013).One area of focus for leadership development in middle school students is leadership selfefficacy which has been described as being crucial for successful leadership (Paglis, 2010).Murphy and Johnson's (2011) model of leadership development includes self-efficacy, motivation to lead, and resilience as attributes necessary for effective leadership.

Instrument Development
Leaders of the STEM-focused curricular enhancement program recruited the authors to identify or develop an instrument to measure innovation and leadership-related dispositions of students participating in a STEM program focused on creating interest in STEM careers.In addition, program leaders were interested in determining if the programs implemented in schools encouraged innovative and creative thinking among students.
Instrument development began by seeking and reviewing instruments that measured innovation and leadership-related dispositions.A review of the existing literature revealed few validated instruments that targeted individuals' attitudes toward innovation, motivation, and leadership.Even fewer of the existing measures were designed for young learners.Not one instrument met the needs of the project team, so the authors developed a tool that included individual items from two validated surveys and created additional items to strengthen motivation and career interest indicators.
Shown in Table 1 are each of the 16 Likert-type items used to create the IAS and their origins.Nine items were selected from the Youth Innovation Skills Measurement Tool (Chell & Athayde, 2009) because they were related to the motivation, leadership, and self-efficacy for being innovative.The Youth Innovation Skills Measurement Tool was created to measure five skills (creativity, self-efficacy, energy, risk-propensity, and leadership) related to innovative behavior matching what employers believe are needed but often lacking.The complete survey tool included 31 items.
Three items were selected from the Entrepreneurial Passion Survey (Cardon et al., 2013) because they related to motivation, enjoyment, and self-idealization of being an entrepreneur.The Entrepreneurial Passion survey included 38 items intended to explore the relationship between entrepreneurial passion and other relevant concepts, including creativity, persistence, and absorption in current activities.Cardon et al. (2013) concluded that passion is at the heart of entrepreneurship and can foster creativity and fuel motivation.
In addition, the authors worked with the STEM program team to add four items to the IAS intended to strengthen motivation and career interest indicators.The authors have developed various attitudinal and STEM career interest instruments (Christensen, 2002;Tyler-Wood, Knezek, & Christensen, 2010;Christensen & Knezek, 2015) and acquired expertise in selecting and shaping items based on the needs of STEM programs.The items were then confirmed by the STEM program personnel as meeting their needs to understand the impact of their program.This process assured initial content validity.

Data Source
Data were collected via an online server by project personnel from 764 students from 12 schools participating in a STEM resource and support program in one U.S. state.All students were in middle/intermediate schools with grade The IAS was administered along with a battery of surveys intended to measure program impacts on students' STEM careers and dispositions.Additional data from one of the surveys, the Career Interest Questionnaire (CIQ; Tyler-Wood, Knezek, & Christensen, 2010) was used in this study to validate the IAS.The CIQ is a five-point Likert-type (1 = strongly disagree to 5 = strongly agree) instrument composed of 13 items on three subscales.The subscales of the CIQ document students' perceptions of being in an environment that is supportive of science careers (Interest), students' intent to pursue educational opportunities that would lead to a science career (Intent), and the perceived importance of science careers overall (Importance).In addition, students were asked to indicate their interest in a career and were able to select from Science, Technology, Engineering, Mathematics, or Other.Responses were dichotomized as STEM or No STEM.

Analysis and Results
Data were gathered in an online system and then prepared and analyzed using SPSS (2010).Descriptive statistics were calculated for each of the survey items and are shown in Table 2.

Instrument Reliability
Cronbach's alpha for the 16-item survey was .91 for this 764-subject data set.According to guidelines provided by DeVellis (2003), the internal consistency reliability would be considered "excellent" and a researcher could even consider shortening the scale (DeVellis, 2003).Because the items came from different surveys, the researchers used factor analysis to determine whether the individual items would align in groups to represent the desired constructs.

Instrument Validation
Measurement specialists (e.g., DeVellis, 2003) refer to reliability (consistency) plus multiple forms of validity (relevance) as being important for a well-constructed survey instrument.While validation of a survey instrument is well known to be an ongoing process, initial indications of validity can be established throughout the instrument development process (Benson & Clark, 1982).Three common forms of validity are content, construct, and criterionrelated validity, the latter of which can be established by alignment with expected measures or demonstration of the ability to separate groups.Construct Validity: Factor Structure An exploratory factor analysis (principal components analysis, varimax rotation) of the 16 individual items on the survey, using the 764 responses, indicated that three constructs were likely well represented by the items on the IAS.Three factors extracted from the data accounted for 57% of the common variance among the responses.These factors were observed by the researchers to measure innovation and leadership attitudes.As shown in Table 3, Factor 1 contained seven items related to being inventive, while Factor 2 contained six items related to motivation and pride in being innovative.Factor 3 included three items related to leadership.

Revalidation of Construct Validity: Multidimensional Scaling
Reconfirmation of the three-construct structure presented in Table 4 was completed using the Multidimensional Scaling (MDS) procedure in SPSS.In MDS, the goal is to determine the smallest number of dimensions that are necessary to accurately represent the psychometric distances between the items rated by survey respondents (Dunn-Rankin et al., 2004).ALSCAL was chosen as the scaling method because it produces an R-squared estimate of total variance explained, which is directly comparable to R-squared values commonly reported for regression analysis.In this study, the specific reason for using MDS was to determine if the clusters of items produced by MDS aligned with the factors produced by factor analysis.
MDS ALSCAL analysis (interval data, Young's S-stress formula, 4 iterations) revealed that when the best-fit of the distances between the IAS items are projected into a two-dimensional (flat plane) space, the result graphically displayed (plotted) in Figure 1 is able to account for 86% (RSQ = .8606) of differences in the ratings among the items, across all respondents.The clusters of items shown in Figure 1 directly correspond to the placement of the items into three factors shown in Table 3.This provides credible revalidation evidence of the three-factor structure presented in Table 3 from a second analysis perspective.

Reliability of Scales Representing Constructs
Factor analysis revealed three scales with Eigenvalues above 1, and these were reconfirmed through multidimensional scaling.Once the three factors were determined, SPSS (2010) was used to compute internal consistency reliability estimates for the scales.As shown in Table 4, Cronbach's alpha was found to be  .88for the seven items loaded most strongly on Factor 1. Cronbach's alpha for the six items on Factor 2 was .81.These are in the range of "very good" according to guidelines by DeVellis (2003).For Factor 3, Cronbach's alpha was .67,with three items in the "minimally acceptable" range (DeVellis, 2003).These values represent the reliabilities (measurement accuracy) of scale scores that could be produced by summing or averaging the items submitted to the SPSS Reliabilities procedure.

Factor 1 Factor 3 Factor 2
When items contributing to the three constructs were averaged into a scale score for each individual, and then the individual scale scores were averaged across all individuals, the result was the mean scale scores shown in Table 5.Note that all score means are greater than 3.0 on a 5-point rating scale, indicating that for this group of students, their innovation-related attitudes were generally positive.

Evidence of Criterion-Related Validity
Criterion-related validity addresses whether the scales tend to separate groups that might be expected to differ (such as males and females) or, conversely, whether the scale values might tend to correlate with other attributes (such as interest in STEM as a career) that might reasonably be expected to align.Analyses of variance and Pearson correlations are techniques commonly used for this type of validation.
The IAS was tested for criterion-related validity by comparing gender, grade level, and interest in a STEM career.As shown in Table 6, separation by the survey resulted in males and females for Factor 3 related to leadership, with females reporting significantly higher (p <.05) scores than males in the leadership area (see Fig. 2).Table 7 includes group mean scores for each of the items on the IAS by gender, with females higher in items related to leadership and males higher in items related to inventing something new.
For the IAS, the 6 th graders were significantly (p <.05) higher than the 7 th and 8 th graders on the complete survey and also somewhat higher on the factors related to innovativeness and leadership (Table 8).However, the survey showed that the greatest discrimination ability by groups tested was for STEM versus No STEM students; that is, those who were interested in STEM as a career versus those who were not interested in STEM as a career.As shown in Table 9, students who indicated an interest in a career in STEM were significantly higher (p < .05) on all three factors as well as the total scale score for the complete survey.A common form of criterion-related validation of a new instrument is to confirm alignment with an established instrument that should theoretically have scales related to newly established measurement indices (DeVellis, 2003).As shown in Table 10, IAS Factor 1 (Inventive) is strongly related to all three scales of the CIQ: Interest (r = .546,p < .01),Intent (r = .535,p < .01),and Importance (r = .537,p < .01).This would be considered a large degree of alignment according to effect size guidelines by Cohen (1988) of r = .1 small, r = .3moderate, and r = .5large.
The IAS Motivation/Pride and Leadership scales were also found to be strongly aligned with measurement scales on the CIQ.In particular, as shown it Table 10, IAS Factor 2 Motivation/Pride was strongly associated (r = .518,p < .01)with CIQ Part 3 Importance, which has an emphasis on making the world a better place.IAS Factor 3 Leadership is moderately associated (Cohen, 1988) with all three CIQ scales of Career Interest (r = .287,p < .01),Career Intent (r = .289,p < .01),and STEM Career Importance (r = .256,p < .01).All of these cross-validation correlation indices fall in the Zone of Desired Effects according to guidelines by Lenhard and Lenhard (2016) and would be considered educationally meaningful according to established research criteria (Bialo & Sivin-Kachala, 1996).

Discussion
One finding in this study, specifically the downward trend for grades 6 through 8 in attitudes toward innovation (Table 8), aligns with similar findings by the authors in the area of creative tendencies.Knezek, Christensen, and Gibson (2022) found that many learning dispositions tend to decline across grades 1 through 12, using data gathered from students in multiple school districts in a midwestern state in the U.S.This implies that elementary school students may somehow have their creative and innovative spirits "educated out of them" in the schooling systems of our industrial age.As stated by Robinson (2006), "we are educating people out of their creative capacities…I believe this passionately, that we don't grow into creativity, we grow out of it.Or rather, we get educated out of it."Therefore, activities that encourage sustaining positive attitudes throughout the elementary and middle school years may be just as important as seeking new ways to foster innovative attitudes at the middle school level.
The positive association of Motivation/Pride with established measures of desire to pursue a career that "makes the world a better place" (Table 10) would also appear to have implications for how best to encourage middle school students to consider careers that tackle societal issues such as climate change.Interestingly, in a prior study with the same CIQ survey, a creative tendencies scale was most strongly associated with two of the CIQ scales: CIQ Interest (interest in a STEM career) and CIQ Importance (wishing to pursue a career that makes the world a better place) (Knezek & Christensen, 2019).These trends are similar to findings from a recent OECD international study noting that a typical teenager today "wants to feel like my work is making a difference and improving the lives of others" (Mann et al., 2020, p. 48).
Significant (p < .001)differences were found on all three IAS scales in favor of students who plan to pursue a career in STEM (Table 9).The magnitude of the difference was large for F1 Inventive, moderate for F2 Motivation/Pride, and still educationally meaningful for F3 Leadership (Cohen, 1988).These findings reinforce the relevance of the IAS scales as potential predictors of STEM career interest.
The significantly (p < .05)higher ratings in this study for females in the area of leadership (Table 6) support the need to provide encouragement and stronger pathways for women to assume more leadership roles throughout schooling while avoiding the tendency of middle school girls to assume the role of 'secretary' in mixed gender small group activities (Eveleth, 2015).The finding also has implications for changes in workforce policy, practice and culture.

Conclusions
Analysis of data from 764 middle school students located in the U.S. indicates that the IAS has respectable reliability as an assessment instrument.Furthermore, factor analysis revealed three constructs that are being measured by the IAS-being inventive, motivation/pride, and leadership.The STEM program for which the survey was created found the data useful as they determined where to target resources and how the IAS indices were related to the goal of creating interest in STEM careers.The IAS should be a valuable tool for educators measuring pre-post changes in middle school students who are part of an intervention or curriculum featuring STEM-related activities.The survey could be useful as a baseline measure for the development of programs to target implementation of innovation-based program elements or to compare subsets of a population such as genders or ethnic groups.Educators may also find this survey to be a useful tool for putting students together into collaborative groups for greater success in working on projects that benefit from positive attitudes related to innovation, such as engineering design projects, other STEM-related projects, or other long-term and/or hands-on projects.Finally, the survey could be used to increase understanding of the relationship among other measures related to STEM career interest and STEM dispositions, as it was used in the current study.Future studies are planned to add items to the leadership scale to strengthen the measure's reliability.

Figure 1
Figure 1 MDS Graphical Representation of IAS Items in 2-Dimensional Space

Table 2
Descriptive Statistics for Innovation Attitude Survey Items(N = 764)

Table 3
Factor Structure for the Innovation Attitude Survey

Table 4
Cronbach's Alpha for IAS Factors

Table 6
Comparison of the Innovation Attitude Survey and Factors by Gender

Table 7
Comparison of the IAS Items by Gender