The Design of an Instrument to Assess Problem Solving Activities in Technology Education

Literally millions of dollars have been spent during recent years to build and equip new or renovated technology education laboratories and to implement contemporary instructional strategies (R. Barker, personal communication, February 14, 1997). For instance, in the state of Georgia alone, over $23.9 million dollars has been spent since 1989 on modular-type programs (Gossett, 1997). Modular curriculum designs have been widely adopted and the integration of math, science, and technology explored. Modular designs typically provide students, working in pairs, opportunities to progress through a series of guided learning activities with an emphasis on problem solving and a hands-on, minds-on approach to learning about technology. Modular lessons are available to address over twenty different specific technological topics and more are being developed on a regular basis. Although adoption of modular curriculum models is one of the most visible trends in contemporary technology education, other significant issues are also impacting the profession. In particular, efforts to equip students with the ability to solve problems, to think analytically, and to apply technical knowledge to real world situations have become integral to technology education (Technology for All Americans Project, 1996). Whether using a modular approach, a traditional unit lab approach, or some other organizational strategy for instruction, problem solving activities are a relevant and important part of technology education. Proponents of adopting modular curriculum programs for technology education have cited numerous anecdotal accounts in support of the value and accomplishments of their programs, but systematic methods of defining and measuring student outcomes have not been sufficiently developed. As a result, the assessment of modular technology education programs and instruction has not been adequately implemented to guide allocation of resources, substantiate curriculum change, and establish the value of these educational activities within the larger educational community. While similar concerns might be raised about other trends in education, the movement toward modular curriculum designs in technology education has been one of the most prominent and significant issues for technology education professionals. __________________________________

solving concepts and strategies as a significant element in curriculum design and implementation.
The task of solving problems can be undertaken in a variety of ways. Problem solving can be approached from simple trial-and-error efforts and range on a continuum to highly complex approaches. Many technology educators espouse the need to create opportunities for students to learn multiple approaches to problem solving with movement toward the development of models to facilitate student growth in strong mental methods of inquiry when solving technological problems (Herschbach, 1989;Hutchinson & Hutchinson, 1991;Todd, 1990;Wicklein, 1993;Zuga, 1989).
One of the aspects that should distinguish technology education from other program areas that address technological content is the integrated study of technological processes, knowledge, and context. In presenting A Rationale and Structure for the Study of Technology, the Technology for all Americans Project (1996) identified these three components as universals for the study of technology. Knowledge related to technology, and processes related to technology, are taught within the context of manipulative activities with information, physical, or biological systems. Hands-on activities are important, but they are not aimed toward development of vocational competencies. They provide a setting for experiential learning related to technology. Of significance to this study, knowledge of technology and manipulative skills related to technology are relatively easy to measure and assess. Use of technological processes, with associated thinking and problem solving skills, is often challenging to measure and accurately assess. Halfin (1973) conducted the seminal research that identified the mental process used by practicing technologists. Beginning with a review of the writings of ten high-level technologists; including persons such as Thomas Edison, Frank Lloyd Wright, and Buckminster Fuller; Halfin identified, and used a Delphi technique to validate 17 processes which were universal to the work of technological professionals. Wicklein (1996) has since undertaken a follow-up study to re-evaluate these processes and further define each of them. In both instances, research was conducted for the benefit of industrial arts or technology education professionals, but the work would be applicable to anyone with an interest in the mental processes used by technologists.

Mental Processes as a Basis for Assessment
The processes identified by Halfin were operationally defined in his work. In addition, Wicklein developed examples for each of the mental processes to more clearly describe their meanings so that each could be discriminated from the other (Hill, 1996). This task was completed following a thorough study of Halfin's work. Wicklein used these operational definitions and examples in developing instrumentation used to re-evaluate the mental processes, but these materials were also a critical element in the development and use of the assessment described here. Table 1 lists the mental processes and operational definitions developed by Halfin for each of the mental processes.
From a practical perspective, the mental processes used by practitioners in technological occupations provide a useful guide for the assessment of instructional activities and content in technology education programs. In some respects, basing curriculum content on mental processes, a relatively constant set of constructs, is more logical than focusing on technological products which are constantly changing. Instructional use of mental processes and product technologies are not mutually exclusive. Technology education inherently includes hands-on experiences with materials and instruction about technical content. When considered in the proper perspective, however, content related to materials and technical processes is characterized by rapid obsolescence while technological mental processes remain relatively stable and continue to be useful for many years. Both should be included in technology education instruction, but the primary emphasis should be placed on the mental processes. Defining the Problem or Opportunity Operationally. The process of stating or defining a problem that will enhance investigation leading to an optimal solution. It is transforming one state of affairs to another desired state. (OB) Observing. The process of interacting with the environment through one or more of the senses. (Seeing, hearing, touching, smelling, and tasting.) The senses are utilized to determine the characteristics of a phenomenon, problem, opportunity, element, object, event, system, or point of view. The observer's experiences, values, and associations may influence the results. (AN) Analyzing. The process of identifying, isolating, taking apart, breaking down, or performing similar actions for the purpose of setting forth or clarifying the basic components of a phenomenon, problem, opportunity, object, system, or point of view. (VI) Visualizing. The process of perceiving a phenomenon, problem, opportunity, element, object, event, or system in the form of a mental image based on the experience of the perceiver. It includes an exercise of all the senses in establishing a valid mental analogy for the phenomena involved in a problem or opportunity. (CO) Computing. The process of selecting and applying mathematical symbols, operations, and processes to describe, estimate, calculate, quantify, relate, and/or evaluate in the real or abstract numerical sense. (CM) Communicating. The process of conveying information (or ideas) from one source (sender) to another (receiver) through a media using various modes. (The modes may be oral, written, picture, symbols, or any combination of these.) (ME) Measuring. The process of describing characteristics (by the use of numbers) of a phenomenon, opportunity, element, object, event, system, or point of view in terms which are transferable. Measurements are made by direct or indirect means, are on relative or absolute scales, and are continuous or discontinuous. (PR) Predicting. The process of prophesying or foretelling something in advance, anticipating the future on the basis of special knowledge. (QH) Questioning and Hypothesizing. Questioning is the process of asking, interrogating, challenging, or seeking answers related to a phenomenon, problem, opportunity, element, object, event, system, or point of view. Hypothesizing is a process of stating a theory of tentative relationship between two or more variables to be tested which are aspects of a phenomenon, problem, opportunity, element, object, event, system, or point of view. (ID) Interpreting Data. The process of clarifying, evaluating, explaining, and translating to provide (or communicate) the meaning of particular data. (MP) Constructing Models and Prototypes. The process of forming, making, building, fabricating, creating, or combining parts to produce a scale model or prototype. (EX) Experimenting. The process of determining the effects of something previously untried in order to test the validity of a hypothesis, to demonstrate a known (or unknown) truth or try out various factors relating to a particular phenomenon, problem, opportunity, element, object, event, system, or point of view. (TE) Testing. The process of determining the workability of a model, component, system, product, or point of view in a real or simulated environment to obtain information for clarifying or modifying design specifications. (DE) Designing. The process of conceiving, creating, inventing, contriving, sketching, or planning by which some practical end may be effected, or proposing a goal to meet the societal needs, desires, problems, or opportunities to do things better. Design a cyclic or iterative process of continuous refinement or improvement. (MO) Modeling. The process of producing or reducing an act, art, or condition to a generalized construct which may be presented graphically in the form of a sketch, diagram, or equation; presented physically in the form of a scale model or prototype; or described in the form of a written generalization. (CR) Creating. The process of combining the basic components or ideas of phenomena, objects, events, systems, or points of view in a unique manner which will better satisfy a need, either for the individual of for the outside world. (MA) Managing. The process of planning, organizing, directing, coordinating, and controlling the inputs and outputs of the system.
A major impediment in past efforts to assess technology education outcomes has been the difficulty of defining and measuring such abstract concepts as technological literacy and problem solving ability. This dilemma has been exacerbated by the mindset of ex-post facto assessment. The typical pattern has been to provide some type of instructional experience or treatment and then to test participants for learning outcomes. New forms of assessment, such as portfolios and journals, are gaining acceptance and are intended to enhance the learning process in addition to providing evidence of learning (Kane & Khattri, 1995;Travis, 1996). Unless actual development of materials is observed in some purposeful manner, however, even portfolios do not provide a means of assessing learning activities as they occur.
Educational research, particularly within the field of cognitive psychology, has provided a theoretical framework upon which to base an alternative form of assessment. In particular, research has shown a clear relationship between teaching behaviors and student achievement (Brophy & Good, 1986). Teaching behaviors include not only the actions of the teacher, but the instructional activities facilitated by that teacher. With this as a premise, alternative forms of assessment can be considered. Rather than only testing after learning activities are completed, assessment can be conducted during the learning activities. Used in conjunction with traditional forms of testing, this form of assessment holds great potential to provide additional feedback regarding the learning process. If outcomes measured in ex-post facto testing are substandard, data gathered during the learning process can be used to analyze the causal factors. In addition, such a tool could assess processes that are not conducive to traditional testing, either due to lack of an appropriate test or because such tests are not sensitive enough to accurately measure the outcomes.
The purpose of this research was to develop and field test a technique for assessing the mental processes used by students as they participate in instructional learning activities in technology education. The technique focused on mental processes used by technology practitioners in their work, as identified by Halfin (1973), and provided an objective measurement that could be used to assess the procedural content during a learning activity. The processes developed in this study can be readily applied to programs that use a modular approach in instruction because of the relatively structured activities included, the focus on problem solving, and a level of student movement that is conducive to observation. The technique, however, could be adapted and used with other forms of instructional problem solving activities. The results would be relevant for any program focused on development of mental processes, of the type used by practicing professional technologists, for problem solving.

Method
The focal point of the assessment tool developed through this research was a measure of the duration and the frequency of selected mental processes necessary for effective problem solving used by students in the completion of technology education learning activities. The tasks necessary to accomplish this included (1) developing a procedure for identifying the mental processes as they were used by students, (2) creating a tool to aid in analyzing the duration and frequency of the mental processes used by students, and (3) testing the system for consistency and reliability.
It is relevant to note that while the term assessment often is used within a context where a value judgement is made and one thing is determined to better than another, the process described in this study uses the term operationally to describe procedures for identifying particular activities, determining how long these activities last, and how frequently activities are repeated in practice. The procedure would enable an observer to determine whether a learning activity accomplished objectives related to use of mental processes in problem solving. It was not, however, designed to directly measure the products or outcomes of the activities involved.
For purposes of this study, the mental processes identified by Halfin (1973) were used. A document was assembled in which a definition was stated and examples were listed for each of these processes. This document provided a ready reference to clarify each of the processes and was frequently referred to during the assessment procedures. Two-letter codes were also developed for each of the mental processes to be used for recording purposes. Table 1 provides a list of the two-letter codes, the mental processes, and their definitions. The document used during the study also included from 6 to 10 behavioral examples for each of the mental processes. It was not included in this manuscript for the sake of brevity, but a copy of it can be obtained directly from the author.
The basic procedure for identifying which mental processes were being used by students consisted of carefully studying the written materials and instructions for a particular learning activity, and then observing students as they completed the activity. Videotapes of students completing modular technology education learning activities were used for the development of the assessment procedure reported in this study. This was necessary to be able to test for consistency and reliability of the assessment procedure. With appropriate instruction and experience, it was anticipated that the assessment procedure could be performed during a live observation session if preferred. It should be noted that it was necessary to view the students themselves, the written materials they were following, the apparatus being used by the students, and to hear the student conversations in order to accurately identify the mental processes being used.
Three pairs of subjects were videotaped for purposes of this study. All students attended a high school or middle school located in the southeastern United States. One pair of male high school students, one pair of male middle school students, and one pair of middle school students consisting of a male and female were voluntary participants for the study. High school students were videotaped as they completed two activities of a construction module prepared by a commercial vendor of technology modules and equipment. The middle school students were videotaped as they worked through two activities of a color computer-aided publishing module produced by a supplier of computer peripherals. These activities covered word processing fundamentals and page layout. The authors who developed the computer-aided publishing module were experienced technology educators with a combined total of 28 years in the profession. The equipment used by the middle school students in the module consisted of a desktop microcomputer, desktop publishing software, and a color ink-jet printer.
To field test the assessment procedure, written materials for the instructional activities were first carefully reviewed and notations were made, using the two-letter mental process codes listed in Table 1, to identify the mental processes students were likely to use at various points during the lesson. The list of mental processes with codes, operational definitions, and behavioral examples was kept at hand during the actual assessment procedure.
Preliminary testing of the observation procedure was done using a timer to record the duration and frequency of each mental process observed. Two independent observers completed two observation procedures each using the videotape of the two high school students working with the construction technology module. This phase of field-testing demonstrated that agreement could be achieved between observers independently viewing videotaped technology education activities. In this initial test, rate of agreement was 100% for identifying the mental processes being used and the duration and frequency measures were within an 80% rate of agreement for the two independent observers. The recording process was cumbersome, and the tabulation of long observations tedious.
In response to difficulties noted in the preliminary testing phase, a computer program was developed to provide a tool for recording and analyzing the duration and frequency of use of the various mental processes by students. The function of the computer was to serve as both a timer and a counter, allowing the observer doing the assessment to simply key in the two-letter mental process codes as they were observed, and having the computer to tabulate the results upon completion. The program, written by the principal investigator of the study and coded in BASIC, was named Observation Procedure for Technology Education Mental Processes (OPTEMP).
The refined assessment procedure was initiated by running the OPTEMP program and selecting the Mental Processes Measurement option from the main menu. After responding to some questions related to observation subject and observer, timing was ready to start. The videotape was turned on and as actions reflecting various mental processes were observed, the two-letter codes were keyed into the computer. With each change of activity, mental process codes were entered. The computer program timed each event, tallied the frequency for each, and following completion of the session provided a printed summary for each mental process code entered.
To establish reliability of the OPTEMP procedure, two observers independently observed and recorded the duration and frequency of mental processes using the three videotaped sessions of middle school students completing activities using the computer-aided publishing module. Prior to beginning the first observations, the modular materials used by the students were reviewed and the observers discussed definitions of the mental processes. Observer 2 found it helpful to classify and label the anticipated mental processes in the instructor copy of the printed materials used by the students. This observer made adjustments and additions as the videotape was viewed, but the initial analysis aided the accuracy of the final observations recorded.
The first OPTEMP observation was completed using videotape of the male and female middle school students completing a page layout activity. The additional observations used videotapes of the male and female middle school students completing a word processing activity and the two male middle school students completing the page layout activity. The typical pattern used by the students as they worked together was for one student to read aloud the instructions and identify the significant steps while the other student worked with the computer. They worked together in this manner throughout each of the activities and discussed the instructions provided in the module as they completed the steps described there.

Results
The summary reports were the key artifacts used in assessing the outcome of the three videotaped field tests analyzed using the OPTEMP. The total duration in minutes and a frequency count was recorded for each mental process as interpreted independently by the two observers. Table 2 provides the results for the first test of inter-rater reliability. Pearson correlation coefficients were calculated to determine how reliable the OPTEMP results were for the two independent observers. The correlation coefficient for overall duration on the mental processes observed in the first inter-rater reliability test was .94 and the correlation coefficient for frequency was .95 (see Table 3). There were two discrepancies in the mental processes coded, with observer 1 coding instances of analysis and observer 2 coding experimenting. In addition, a 2.39 minute difference in total duration for the procedure indicated a need for better cueing of the videotaped session.
Results from the second inter-rater reliability test are presented in Table 4. The correlation coefficient for duration in this test was .88 and the correlation coefficient for frequency was .92. Two discrepancies in mental processes occurred with observer 1 recording experimenting and observer 2 recording instances of creating. Overall durations for the two observations were approximately equivalent for this session.
The results for a third inter-rater reliability test produced a correlation coefficient of .93 for duration and a correlation coefficient of .91 for frequency. The observers were in agreement concerning the mental processes that were observed (see Table 5) and the overall durations of the observations were suitably close.
Observer 1 completed a test and re-test for the videotaped session used in the first inter-rater reliability test, the tape of male and female students completing the page layout activity. The results of this test are presented in Table 6. The correlation coefficient for duration in this repeated test was .95 and the correlation coefficient for frequency was .97. One discrepancy in mental processes occurred with instances of experimenting being observed during the second viewing of the videotape. In addition to the numerical results reported in the tables, several conclusions were noted regarding the OPTEMP system. Both observers found that use of the procedure caused them to analyze what was happening during the technology education instructional activities in a more detailed manner than they had done so before. They also noted that the nature of the modular curriculum materials aided the process because the observer could become familiar in advance with the basic path the students would follow and this facilitated identification of most of the mental processes that would be used by students.
The overall agreement among independent observers regarding which mental processes were being used provided some evidence that the OPTEMP was valid. To provide additional evidence of validity, feedback from students about their own interpretation of the mental processes being used during technology education activities could be obtained and compared with the OPTEMP results. This technique was precluded in the present study due to the young age of the middle school participants and their limited understanding of the mental processes as defined, but upper level high school or post-secondary students would be capable of comprehending and distinguishing their own use of the mental processes.

Conclusion and Recommendations
Based on the inter-rater reliability test results and the repeated test results, the OPTEMP was determined to be an effective tool for assessing the use of mental processes during completion of modular technology education learning activities. The data gathered using this technique would enable an instructor or researcher to accurately determine which mental processes, used by practicing technologists, were being implemented by technology education students. Activities could be modified and further assessed to incorporate key processes not being used and changes could be made to adjust the duration and frequency of processes presently in use.
The use of modular technology education instructional activities in the development and testing of the OPTEMP facilitated this study because students working in that environment were readily available, limited physical movement facilitated videotaping, and instructional content incorporated numerous instances for problem solving. The specific comments provided in the remainder of this section reference the use of modular curriculum materials, but these are not intended to preclude the use of the OPTEMP with other instructional formats.
These comments are also based on two key assumptions. The first is that duration and frequency of using a mental process are related to learning outcomes specific to that process. In other words, the more someone uses a mental process, the better one will be at using that mental process. The other assumption is that it is appropriate for technology education to facilitate student development of mental processes used by practicing technologists. Halfin (1973) and Wicklein (1993Wicklein ( , 1996 have provided a sound rationale for this.
The modular technology education lessons used in this study were not purposefully designed to provide students with opportunities to apply the mental processes identified by Halfin. As is the case with all well-designed technology education modules, problem solving activities were included. Incidental to that, uses of mental processes were involved in the work necessary to complete the instructional activities. The OPTEMP has significant value for technology education professionals who intend to encourage development of mental processes in some deliberate way. By providing a tool for assessing the process content of modular learning activities, the OPTEMP facilitates comparison of the various instructional products on the market and makes possible more informed choices about use of scarce educational resources.
Further development and revision of previously installed technology education modules is another area of concern for most practicing technology educators. These changes and enhancements are typically guided by problems such as apparatus not working correctly, student confusion about instructions, or time requirements for an activity. Use of the OPTEMP could further enhance revisions of existing materials. By identifying mental processes with less duration and frequency of use, revisions could be directed to provide problem solving experiences that include a more balanced treatment of the mental processes used by technologists.
The OPTEMP could find meaningful use in the preparation of technology education teachers. In considering curriculum design and other related issues, the observer using the OPTEMP would gain a heightened awareness of what is really happening for students as a learning activity is conducted. Just as a color or other characteristic of an object can go unnoticed unless attention is called to it, student learning activities can be observed but not really understood in the absence of a systematic approach for analysis. Use of the OPTEMP during curriculum classes, intern experiences, and perhaps as a tool during student teaching could enhance the preparation of future technology education teachers. Wicklein (1993) has proposed development of a process-based technology education curriculum. In such a design, technology education instructional activities would be organized around the mental processes of technologists rather than around a product-based technology. Content related to communication, transportation, manufacturing, and construction would still be included, but the arrangement of this content would differ. Goodlad (1966) recommended designing curriculum that had a systematic, carefully considered approach, but not necessarily a step-by-step sequence of topics in ascending order of difficulty. Structuring curriculum around the mental processes would emphasize the more permanent constructs associated with thinking and problem solving. Technical content would be included, but specific skills and knowledge of materials would be presented with the understanding that changes could be quickly expected. Emphasis on the relatively stable mental processes would provide learning experiences that would have lasting value for students.
For those that would develop and adopt a process-based approach to curriculum, the OPTEMP provides a readily available assessment tool for analyzing the results. Instructional materials designed around traditional approaches to technology education can be evaluated, but the instrument aligns with the stated purpose of process-based materials and would therefore provide a more accurate assessment of their effectiveness.
In this initial work with the OPTEMP, the focus was directed more toward the design of the learning activities than on individual learner performance. Further research is needed to explore uses of the OPTEMP for assessment of learning outcomes. At present no claim has been made other than that OPTEMP scores indicated the duration and frequency of mental processes that students used as they went about the problem solving activities included in technology education learning activities. It is anticipated that correlation between OPTEMP scores for student use of appropriate mental processes and gain scores on established tests would be moderate to high. Those students who spend time off task or who make significant use of inappropriate mental processes would be expected to be less efficient in their learning about technology. Use of the OPTEMP in conjunction with traditional tests might produce a diagnostic assessment which would aid in helping students develop more effective work habits and learning strategies.
Additional testing of the OPTEMP beyond this initial study is also needed to address issues of variability. Use of the procedure by a greater variety of observers and with different types of students, perhaps with differing cultural backgrounds, is requisite to further establishing reliability and validity. In addition, further testing with several additional types of modular technology education activities, and perhaps with other instructional strategies that incorporate problem solving, would determine the versatility of the procedure. The study described here has provided a description of the procedure and offered results of initial testing, but much additional work is needed to verify the usefulness of the OPTEMP and to further refine procedures for its application.
The issue of assessment will be of critical importance to every area of the educational enterprise as the new century dawns (Stiggins, 1995). The assessment system developed and tested in this study was shown to hold promise as a reliable and useful tool for analyzing important components of technology education problem solving activities and should be of benefit to the profession. The potential benefits range from aiding the identification of quality instructional materials to assisting in the preparation of technology education teachers. As previously mentioned, work is needed to further refine the OPTEMP procedures, to enhance the computer program, and to establish validity and utility of the system. Without adequate assessment procedures, technology education cannot reach its full potential and it will continue to struggle for recognition and acceptance within the greater educational community.