Self-Report and Academic Factors in Relation to High School Students’ Success in an Innovative Biotechnology Program

Biotechnology constitutes one of the most challenging, cutting-edge, and rapidly growing fields in science today (U.S. Department of Labor, 2008). Its products influence our daily lives on multiple levels and often improve our quality of life. Both the practical implications and the hands-on nature of this ‘modern science’ make the topic of biotechnology an attractive addition to the high school science curriculum (France, 2007). The interdisciplinary nature of biotechnology also makes it an ideal candidate for future curricular offerings that strive to incorporate the K-12 Framework for Science Education (National Research Council [NRC], 2012) and the Next Generation Science Standards (NGSS; http://www.nextgenscience.org/). Bybee (2011) noted that the shift from inquiry methods to the use of both science and engineering practices will likely be the greatest challenges related to the NGSS, and suggested that these practices should be thought of as both instructional strategies and learning outcomes. Indeed, the NGSS performance expectations place emphasis on combining practice and content in the assessment of student learning (Bybee, 2012). Sneider (2012) suggested the use of existing learning activities as a first step in integrating technology and engineering practice into science teaching. The literature provides both a conceptual base for integrating biotechnology into curricular offerings, as well as practical examples implemented by early adopters. Wells (1994), for example, gathered biotechnology experts to compose a common taxonomic structure to guide the development of high school biotechnology curricula. Twenty panel members identified eight main areas, including topics that span both science (e.g., biochemistry, medicine, environmental science) and engineering (e.g., genetic engineering, food science, environmental safety). Similarly, an early survey of biology teachers captured recommendations for science and engineering content (the structure and function of DNA, understanding the genetic code, genetic engineering, cloning, and the biology of cancer), as well as the recommendation that this content be delivered through labs when possible (Zeller,1994). Harms (2002) noted that providing students with the opportunity to practice biotechnology provides specific examples of applied science that generalize to a differentiated understanding of concepts.

A small number of studies have been published to describe the impact of biotechnology programs (Bigler & Hanegan, 2011;Gabric, Hovance, Comstock, & Harhisch, 2005;Marchaim, 2001;Powell & Stiller, 2005;Santucci, Mini, Ferro, Martelli, & Trabalzini, 2004).Initial results are positive, indicating an increase in student interest and a positive shift in motivation to learn (Powell & Stiller, 2005;Santucci et al., 2004).Knowledge gains have also been reported, though further investigation indicates that knowledge has been measured broadly.Dawson & Soames (2006), for example, asked students to list as many examples as possible in relation to biotechnology, genetic engineering, and cloning.Correct examples were scored as indicators of knowledge.Pre-post assessments have also been used, as well as post-only open-ended questions (Mueller, Knoblock, & Orvis, 2009).The latter method, in particular, indicated that students' ability to apply biotechnology knowledge was significantly improved if teachers used active rather than passive teaching techniques.Additional research with pre-post pencil-paper assessments suggests that handson classes that utilize professional equipment enhance student learning above and beyond the gains normally associated with these classes (Bigler & Hanegan, 2011).
Framed within the context of the NGSS, these descriptions indicate that many biotechnology programs are providing students with opportunities to engage in science and engineering practice and that these experiences result in positive outcomes.The measures used in the literature to date have been specific to biotechnology rather the broader science and engineering practices embedded within the activities.None measured skill and few measured applied knowledge within the context of practice as envisioned by the NGSS.Assessment of these experiences may be a challenge given that it is difficult to assess student understanding of the various topics introduced via hands-on activities (Steck, DiBase, Wang, & Boukhtiarov, 2012).As we move into the next generation of science standards, it will be critical to integrate the use of biotechnology techniques and equipment within the disciplinary core ideas of the NGSS, and to provide appropriate measures to document student learning within this context.

Background and Purpose
The need for a shift in science education that moves from the memorization of facts to learning by doing has been widely recognized for decades and now has the potential to be realized through the NGSS (National Science Teachers Association, 1982;National Science Board, 2007;[NRC], 2012).One innovative program poised to respond to this shift is the UCSD ScienceBridge Tech Sites.With funding from the Howard Hughes Medical Institute (#51006101) and the United States Department of Education (#R305A080692), ScienceBridge has pilot tested the use of high school-based biotechnology production sites.Within this context, biotechnology is defined in relation to specific tasks related to genetic engineering, protein structure and function, and enzyme reactions.There are eight departments in the Tech Site model: Solutions and Aliquots, Microbial Media, Inventory Management, Risk Management, Facilities Management, Order Fulfillment, Customer Service, and Quality Assurance.Working within and across departments students engage in tasks such as creating solutions, maintaining live cultures, managing inventory, and distributing materials.The balance between entry level jobs and management positions that require more extensive scientific and technical expertise also reflect the biotechnology industry.Students can move from team to team across multiple semesters to acquire a well-rounded skill set.
Tech Site courses were offered to students through a district-level Regional Occupational Program (ROP) designed to provide career skills to students through community partnerships (Mitchell, Adler, & Walker, 2011).Though implemented as a technology education course, the Tech Site's biotechnology focus is also appropriate for science and engineering courses.The NGSS Scientific and Engineering Practices embedded in the experiences include Asking Questions, Developing and Using Models, Planning and Carrying out Investigations, Analyzing and Interpreting Data, and Constructing Explanations and Designing Solutions.The science content largely focuses on Disciplinary Core Ideas within Life Sciences (LS1, LS2, LS3, and LS4) as well as Engineering, Technology, and Applications of Science (ETS2).Working collaboratively to manage kit production and delivery reiterates several additional NGSS ideas, including Connections to Engineering, Technology, and Applications of Science, the Nature of Science, and Science is a Human Endeavor.
Evaluation of the Tech Sites began in 2008 using a pre-post survey that was administered to students at the beginning and end of the course.In later years, student academic data were also gathered.The Tech Site evaluation is best categorized as a developmental evaluation, an approach that collects data to guide adaptation to the emergent and dynamic environments that occur during the development of projects, programs, and/or policy reforms (Patton, 2011).The Tech Site evaluation is considered developmental because the data were collected as the project evolved in response to changes in the overall ScienceBridge program.For example, ScienceBridge created new biotech labs each year, thereby expanding the skill set required of Tech Site students.The individual Tech Sites also developed in nuanced ways based on the culture of both the individual school and district.The evaluation was also conducted at the crossroads of a policy change ([NRC], 2012), and thus the data have the potential to serve as a developmental evaluation of indicators and methods that might prove useful to those interested in implementing biotechnology courses in response to the NGSS.This exploratory study investigates two questions: 1. What self-report measures can be used to document success in a high school biotechnology course?
2. What is the relation between self-report measures and more traditional forms of assessment, and can self-report measures be used to predict course success?

Method Participants
Students from three high schools in two urban districts participated in the Tech Sites from 2010-2012.All students participated in the evaluation each year.In total, Tech Sites served 183 students over the course of this two year period; 178 participated in the evaluation (97%).Demographic characteristics are presented in Table 1.There were more females than males and over half of the students identified as Hispanic.

Measures
Pre-and post-surveys were administered to students online at the beginning and end of each term, respectively.One Tech Site allowed students to enroll for multiple terms; for these students, their initial pre data were used in comparison to post data from their final term of participation.Three survey constructs were of interest for the current study: students' attitudes toward science and both their self-reported awareness of and proficiency with biotechnology skills.Attitude was measured via a battery of survey items that differed each year (see the Appendix for survey scales and items).In 2010-2011, the science items from the STEM Semantics Survey were used (modified from Tyler-Wood, Knezek & Christensen, 2010).The battery consisted of five adjective pairs.Each pair included one negative and one positive adjective and a series of six open boxes that connected the two terms.Students were asked to choose the option between the adjective pair that best reflected their opinion of science.In 2011-2012 the Math Science Interest Scale (Fouad & Smith, 1996) was used to measure students' interest in 20 specific science or math related activities by asking them to rate their interest on a five-point Likert scale: Strongly dislike, Dislike, Somewhat like, Like, Strongly like.To equate the different attitude ratings scales used, students' average scores on the five-point scale were multiplied by 6/5.
To measure students' awareness of biotechnology skills at the beginning of the Tech Site course, students rated a list of activities using three options: I had never heard of this skill before I started the Tech Site course; I had heard of this skill before starting the Tech Site course, but I had never used it myself; I had used this skill before I started the Tech Site course.Fifteen specific skills were rated in 2010-2011; six new items were added to the list in 2011-2012 to reflect new areas of focus within the program, bringing the total to 21 items.Using a different scale, students rated their proficiency with these skills at the beginning and end of each term: I cannot perform this skill; I can perform this skill but only with assistance; I can perform this skill well enough to do it on my own; I can perform this skill pretty well and could teach it to a friend if I had time to review; I can perform this skill very well and could teach it to a friend right now.
Three academic measures were used in the study.Students' scores on the California High School Exit Exam (CAHSEE) were used as a measure of baseline English/Language Arts (ELA) and Math ability (for reliability and validity information see California Department of Education, 2012).This statebased exam was created to ensure that students graduate with basic skills in each discipline.Students take the exam for the first time in 10 th grade and then again in subsequent years if they do not achieve a passing score.For the purposes of the current study, students' final scores in both ELA and Math were used.
The second academic measure was students' final course grade, which included classroom participation in biotech production activities as well as scores on written and practical exams.The third academic measure was the score on an articulation exam offered by San Diego Miramar College's biotechnology program.Two of the Tech Sites were part of a college program that allows students to earn college course credit for their high school biotechnology class.All students with a course grade of B or higher in an approved class were eligible to take the written exam and those who pass the exam receive four units of college credit.The exam measures general biology/biotechnology content knowledge, understanding of computational/data analysis content, and understanding of important biotechnology techniques.Correlation coefficients were used to investigate the relation among key variables.Using the Bonferroni approach to control for Type I error across the 36 correlations, a p-value equal to or less than .001(.05/36 = .0014)was required for significance.The results of the correlational analyses presented in Table 4 (next page) show that 13 out of 36 correlations were statistically significant.Moderate to high positive relations were found between all academic measures.There was no consistent pattern in how attitude related to the other variables.Skill at the end of the course was positively related to a number of variables, including a moderate relation to course grade and a high relation to exam score.

As shown in
Two path analyses were also conducted, one for course grade and the other for exam score.Path model analyses were run using Mplus, Version 7. The following equations and Figure 1 describe the initial path model.Full information maximum likelihood estimators of parameters were used.Students' ELA and Math scores and post attitudes had a statistically significant association with their course grade ( 3 = 0.091,  < .05; 4 = 0.065,  < .05; 9 = 3.536,  < .001).Post skills were significantly predicted by students' pre awareness ( 1 = 0.386,  < .01)and pre skills ( 2 = 0.244,  < .01).Post attitude was significantly predicted by pre attitude ( 3 = 0.648,  < .001).Further, pre attitudes had a significant indirect effect on course grade through post attitudes ( 3 ×  9 = 2.291,  < .01).Model fit statistics also indicated a good fit of the model to the exam score outcome.RMSEA=0.021,CFI=0.993, and SRMR=0.033.Table 7 presents the direct effects and Table 8 (next page) presents the indirect effects.Students' Math score and post skills had statistically significant associations with their exam score ( 4 = 0.263,  < .01; 8 = 5.493,  < .05).Post skills were significantly predicted by pre-awareness ( 1 = 0.415,  < .01)and pre skills ( 2 = 0.236,  < .01).Post attitude was significantly predicted by pre attitude ( 3 = 0.647,  < .001).None of the pre measures had a significant indirect effect on exam score through post attitude or skill.

Conclusions
This study was part of a developmental evaluation of the UCSD ScienceBridge Tech Site program and explored the relationships among selfreport and objective academic measures.As with any exploratory research, the purpose of the current study was to understand a phenomenon in greater detail with the hope of identifying hypotheses that can be explored in future research.
Documenting pre-post change on key variables of interest is a common method of measuring impact in program evaluation studies, and this method was used throughout the first years of the Tech Site evaluation.Course success in this context was defined as a statistically significant increase in students' scores after the program.The current analysis indicates that the UCSD Tech Sites were effective at improving the biotechnology skills of students, while science attitudes remained unchanged.
Initial studies of high school biotechnology courses have suggested that student motivation and knowledge are impacted positively and that impact is most pronounced within the context of active learning strategies (Bigler & Hanegan, 2011;Powell & Stiller, 2005;Santucci et al., 2004).The current study is the first of its kind to explore how student characteristics and biotechnology course outcomes relate to one another.Results indicated that students' general attitudes about science and their overall math skills predicted success across a number of measures, including both self-reported skill ratings as well as the more traditional measures of course grade and exam score.Further, the results demonstrate the association between skill, as measured through self-reported proficiency ratings, and achievement on an articulation exam.This link was particularly important for the program given that it was operated within the context of an ROP; it documented the relation between skills students learned in the course and the potential next steps in their educational and/or technical careers.

Implications
The results from the current study offer additional points of both intervention and evaluation for educators implementing biotechnology courses.The association between attitudes and knowledge in the literature includes a moderate correlation found among Israeli high school graduates (Pe'er, Goldman, & Yavetz, 2007), as well as a weak correlation found for secondary students from five European countries (Dijkstra & Goedhard, 2012).The results from the current study add to this literature by demonstrating that science attitudes may serve as a constant in relation to students' success in science courses.Science attitudes were predictive of academic outcomes despite the fact that there were not changes in attitude scores from pre to post.Ayendiz and Kaya (2012) concluded their study of Turkish high school students' science attitudes by stating that "classroom instruction has a significant impact on students' attitudes toward science (p.44)."It is important to note that each of these international studies positioned their results within the context of either the educational and/or cultural system being studied.Future research should investigate the relation between attitude, attitude change, and academic outcomes among American students.Differences in how these variables interact in the context of biotechnology, engineering, and science courses should also be investigated.Should this research confirm that attitude plays a role in students' academic outcomes it would provide an additional leverage point for educators to utilize as they work to impact student learning.
The results from the current study also have implications for how student learning is measured.Norton (2004) suggested that teachers use formative assessment as a tool to allow students to collaborate with others and express ideas in their own unique and creative ways.The results from this study indicate that self-reported proficiency measures might serve as an accurate formative measure when data are being collected on specific skills.Students' proficiency ratings increased across the life of the Tech Site, were correlated from the beginning to the end of the course, and post proficiency ratings also correlated with both course grade and exam scores.The proficiency ratings conducted as part of this evaluation are also practical for today's biotechnology classroom; they were quick to administer and required no additional materials or resources.
It is somewhat surprising that students' self-assessments were also predictive of exam scores.As researchers begin to focus on ways to measure students' proficiency with science and engineering practice in relation to the NGSS, it may be useful to consider whether and how self-assessment of these skills provides a valuable measure of student learning across disciplines.Future research should validate the utility of this idea by comparing students' self-reported skills with those measured through objective performance-based assessments such as practical exams.Proficiency skill ratings may prove a valuable response to the challenges associated with assessing lab-based, problem-based, and hands-on learning experiences (Savin-Baden, 2004;Steck et al., 2012) at a time when these learning experiences are likely to become more rather than less common.
The topic of biotechnology has been a new source of study within recent years as educators and researchers grapple with ways to achieve and measure student learning in relation to this modern science.The thrust of the NGSS aligns nicely with the topic of biotechnology and provides a host of new challenges related to establishing integrated curricular materials and assessments that move beyond content to measure science and engineering practice.Whether biotechnology courses continue to be implemented within the context of technical programs such as ROP, as engineering programs, or by science educators trying to align with the NGSS, the results from this exploratory study document student characteristics to keep in mind as educators and policymakers consider whether and how to incorporate biotechnology into high school curricula.This study also highlights potential measurement strategies for educators and researchers to use to understand student learning in relation to this content.

Table 1
Table 2, internal consistency estimates of reliability were computed for each construct on the student survey.The values for Cronbach's alpha ranged from .84 to .96across constructs, indicating satisfactory reliability for each variable of interest.Based on this consistency, students' average score across items was used for further analysis of each construct (i.e., Attitude, Awareness, and Skill).

Table 2
tests were conducted to evaluate whether students' science attitudes and biotechnology skills improved after the program.As shown in Table3(next page), results indicate that attitudes remained constant before and after participating in the Tech Site.Students' biotechnology skills improved significantly after the program.