ASSESSMENT OF ACHIEVEMENT IN PROBLEM-SOLVING SKILLS IN A GENERAL CHEMISTRY COURSE

This artcle reports the development and validaton study of tests to assess achievements at three levels of knowledge structure, following the model proposed by Sugrue to measure problem-solving skills. The literature has reported this proposal as a model consistent with the theoretcal constructs underlying problem-based learning (PBL) methodology. The tests were constructed for a General Chemistry course in a curriculum of engineering, which implements PBL methodology at a Peruvian university. The content validaton of the tests was performed, as well as a pilot implementaton with Peruvian students of frst year engineering. The results obtained in omissions percentage, difculty degree, items response patern and the point biserial coefcient (rpb), let us to conclude that these are appropriate tools for assessing these skills, mainly for the purpose of contributng to process facilitaton as well as to future research in this line.


INTRODUCTION
Despite the variety of approaches to problem-based learning, in terms of its defniton and diferent modalites of implementaton (Dochy, Segers, Van den Bossche & Gijbels, 2003;De Graaf & Kolmos, 2003), there is no doubt that there is a consensus to point out that from the cognitve dimension, one of the main objectves in this educatonal proposal is the development of problem solving skills both for the acquisiton of knowledge, as for its applicaton (Hmelo-Silver, 2004).Although there is a large volume of research on PBL, there is no consensus on what achievements or what measurement methods should be used to evaluate their efects, and this situaton leads to the need to identfy relevant theoretcal constructs underlying PBL features and main goals in the learning context (Newman, 2003).The meta-analysis performed by Gijbels , Dochy , Van den Bossche and Segers (2005), used Sugrue's model (1994Sugrue's model ( , 1995) ) as a frame of reference to study the PBL efects when the assessment of its main goals focuses on the understanding of concepts, principles that link concepts, and the linking of concepts and principles to conditons and procedures for applicaton.For the authors, the great advantage of Sugrue model is that it is translated into specifcatons for the assessment of the main cognitve components of problem solving: knowledge structure, metacognitve functons, and motvaton.
This study follows the line of work of Gijbels et al. (2005), to develop instruments with the purpose to monitor student achievement in three levels of knowledge structure using Sugrue model, in a General Chemistry course where a hybrid PBL approach was implemented.

PBL theoretcal framework
PBL methodology is part of an educatonal vision that promotes an open, critcal, and refectve learning with a holistc approach to knowledge that recognizes their complex and changing nature, and involves a community of people who interact collaboratvely to make decisions regarding diferent problem situatons they must face.PBL, in this sense, is the medium to set the conditons to promote actve, contextualized, integrated and comprehension oriented learning, providing opportunites to refect on the educatonal experience and to practce the applicaton of learning (Margetson, 1997;Engel, 1997).Therefore, it is clear that the learning goals go far beyond the mere acquisiton of disciplinary content.
The scenario or problem that initates the learning process fulflls several functons during the process: to organize the content and knowledge; provide the context of the learning environment, stmulate higher order thinking and refecton, as well as promotng and maintaining motvaton for learning.Hung (2006) proposes an interestng model that clearly illustrates the diferent components of the scenario or problem design, and can be a reference to identfy the main issues involved in PBL learning goals.
The core components of the model: content, context and connecton are mainly related to the ownership and adequacy of content knowledge, and their contextualizaton and integraton.The content, as a central component, validates proper alignment between the problem scope and the curricular contents (expected learning) in breadth and depth.The context ensures that the situaton presented is authentc (real or realistc), relevant to the professional area in which the student is formed, their personal interests or his life as a member of society.The connecton component enables students to integrate knowledge and interconnect the concepts in such a way of establishing a conceptual network of the subject.
Processing components: research, reasoning and refecton facilitate conscious and meaningful involvement of students in their learning process.They are the dynamic elements of the model, thus, research enables the core components, promotes the development of skills for the search and processing of informaton, facilitates conceptual connectons for student learning and supports the processes of reasoning and refecton.The reasoning component promotes the development of skills of higher order thinking, actves core components also facilitates student conceptual connectons and supports the processes of research and refecton.The later component acts as a guiding element for metacogniton, facilitates refecton process to synthesize and integrate the knowledge learned, promotes in students the habit to develop and use their skills to self-directed learning and lifelong learning.
There is an additonal important element to consider: PBL process is developed on the basis of efcient teamwork dynamics.The infuence of interpersonal relatonships and communicaton with others about learning is recognized in both learner-centered psychological principles proposed by APA (1997), and the constructvist view of teaching and learning (Coll, 2001).Working collaboratvely with peers favors the actvaton of prior knowledge, provides opportunites for exchange of ideas contributng to the restructuring of paterns of knowledge, stmulates epistemic curiosity, attude and motvaton for learning (Schmidt, 1983).
According to discussed so far, PBL methodology helps students to construct an extensive and fexible knowledge base; develop efectve problem-solving skills; develop self-directed, lifelong learning skills; become efectve collaborators; and become intrinsically motvated to learn (Hmelo-Silver, 2004).
The adopton of PBL involves substantal changes in various aspects related to insttuton, teachers and students, which will ultmately determine the achievement of the fundamental goals of this educatonal proposal.This transiton is not a simple process and therefore, it has emerged a variety of approaches to the methodology ranging from adopton in isolated courses belonging to a traditonal curriculum, untl joining a radical curricular transformaton (Dochy et al., 2003).The diversity of approaches has hampered the interpretaton of results of research on their efectveness (Camp, 1996), which is why from the late ninetes, several studies have recognized the need to consider the characteristcs of the specifc context in where the PBL implementaton is performed, in order to have more and beter informaton for evaluatng the achievements of the methodology and the factors that may afect them (Dochy et al., 2003;Gijbels et al., 2005;Neville, 1999;Newman, 2003).

PBL approach implemented in the study
In Latn America Engineering Programs the more frequent experiences correspond to PBL approaches, mainly hybrid models that incorporate diferent kind of variatons of the original model, implemented in one single course.In the case of the General Chemistry course were this study was performed, each unit was organized as shown in Figure 1.

Figure 1. Hybrid PBL approach implemented in the General Chemistry course
The assessment of each unit considers the proposal of soluton for the PBL scenario, collaboratve learning actvites performed with mediaton of the Professor, the electronic portolio and two individual tests: one of multple optons to evaluate achievements in the three levels of the structure of knowledge according to the model of Sugrue, and other of open response relatng to the contents of the unit.

Problem solving skills in Sugrue model
The literature on problem solving skills is characterized by displaying multple theoretcal frameworks from cognitve science or informaton processing.However, comprehensive models of its components have been proposed based on the review and compilaton of results from several research lines.Sugrue examined common issues of some of these models, in order to identfy a set of cognitve components that could be measured to estmate the extent to which a student can solve problems within a domain, such as science, as well as provide a basis for selectng a subset of variables to be submited for evaluaton.
The three cognitve components to be assessed in problem solving following Sugrue model are: • Cognitve Functons, component that supports fexible adaptaton of self-knowledge to meet the demands of a new problem.It is related to the so-called metacognitve functons, or processes of higher order thinking, • Beliefs, component that aims to generate a comprehensive profle of the student's ability and willingness to solve problems in a partcular domain, through variables: self-efcacy , percepton of task demands and perceived atractveness of the task, • knowledge structure, this component consists of three levels that, in a perfect performance, show a high degree of connectvity, integraton and consistency.
Sugrue suggests that the ability to solve problems in a partcular domain results from the complex interacton of the structure of knowledge, metacognitve functons and motvaton.The diferences observed during the process, from the interpretaton of the problem to the persistence in trying to solve it, can be atributed to variatons in these three aspects of cognitve constructs.For each of the three categories of cognitve components, Sugrue describes a limited set of variables that should be targeted by assessment under two criteria: they were shown to be critcal by research or open to instructonal interventon.In this study, interest has focused on developing tools to assess achievements in the component knowledge structure, so this theme is developed in more detail in the next secton.

Knowledge structure assessment
This component of Sugrue model considers three levels: • First level: Concepts.A concept is a category of objects, events, people, symbols and ideas that share • Second level: Principles.A principle is defned as a rule, law, formula or statement that characterizes the relatonship (ofen causal) between two or more concepts.The evaluaton of the understanding of principles involves: selectng best predictons or best explanaton (in a multple choice format), or generatng predictons or solutons (in an open-ended format), or giving an explanaton of an event or result (in a hands-on format).
• Third level: Linking the concepts and principles to conditons and procedures for applicaton.To facilitate the resoluton of the problem, the concepts and principles should be linked to the conditons and procedures that facilitate their use in new situatons.A procedure is a set of steps that can be done to achieve a goal.The conditons are aspects of the context that indicate the existence of an instance of a concept, or to indicate that a principle is operatng or can be applied, or that a partcular procedure is appropriate.Subjects with good performance in problem solving should be able to recognize situatons where procedures can be performed to identfy or generate instances of a concept and should be able to carry out these procedures exactly.Overall, they should be able to assemble a procedure based on a principle to construct a desired achievement in a new situaton.The evaluaton of this level involves: selectng correct procedure for identfying instances of a concept, or selectng most appropriate procedure to change the state of one concept by manipulatng other (in a multplechoice format); or generatng a procedure for identfying instances of a concept, or generatng a procedure to change the state of one concept by manipulatng other (in a multple-choice format); or performing procedures to identfy instances of a concept, or performing a procedure to change the state of one concept by manipulatng other (in a hands-on format).
The diagnostc evaluaton of problem solving skills should allow the identfcaton of students who understand the concepts but not the principle that binds them, students who understand the principles and concepts but have no knowledge of the procedures for applying them, and students who are able to perform procedures correctly but do not know when it is appropriate their applicaton.
The fndings of Gijbels et al. (2005) showed no signifcant diferences in the frst level, corresponding to the understanding of concepts, comparing the achievements reported in PBL implementatons with conventonal teaching methodology.However, in the second level, corresponding to the understanding of principles linking concepts, PBL showed signifcant superiority.There were not enough studies on the third level of the model, for this reason, previous research aimed to develop tests to assess achievements in the third level that were applied to freshmen engineering courses, in the context of General Chemistry and General Physics courses (Morales 2011a(Morales , 2011b)).
The aim of the present study is to develop and validate instruments that can be used for the assessment of cognitve achievement in the three levels of knowledge structure, based on a model that is consistent with the learning environment promoted by methodologies such as PBL.

METHODOLOGY 2.1 Instruments constructon
The frst task in the constructon of the tests was to identfy the concepts, principles and the conditons and procedures that were to be evaluated in the main thematc units of General Chemistry 2 course.To do this, it was conducted a content analysis of topics and issues in the curriculum, to detail the domain and sub-domain of interest and decide the level of assessment.Then the items were writen taken as reference Sugrue model for evaluatng three levels of Knowledge Structure: concepts, principles and link the concepts and principles to conditons and procedures for applicaton, resultng in the constructon of three tests, one for each topic: Thermodynamics, Kinetcs and Chemical Equilibrium.The format used for all the tests was multple choices.
The items were formulated in such a way that for the frst level, the student should identfy examples of the concept.At the second level, the student should select the best explanaton of a partcular event and in the third level, the student must select the correct procedure to identfy concepts in a given situaton, or select the most appropriate procedure to change a concept status manipulatng another concept.
Some examples: • First level: The soldiers ofen bring prepared food placed in a closed plastc bag or aluminium foil, which can be heated by immersing it in boiling water or by placing it on the hot engine of a vehicle.Another possibility is to use the "fameless heaters," which utlize a reacton between magnesium metal and water, which is very exothermic.
From the text above: Identfy a thermodynamic system and indicate its type (open, closed, isolated), justfying your answer.
• Second level: It is suspected that the decompositon of a biological substance follows a frst order kinetcs.To confrm this, the concentraton of the substance was measured at diferent tmes.What must we do with this data?
• We must determine the concentraton which decomposed, as the diference between the inital concentraton and the remaining concentraton at each tme.If the graph of the obtained values versus tme is a straight line, then it is a frst order reacton.

Content validaton
This is the most important stage in the validaton process, as it shows the logical relatonship between test items and content and skills assessed.For this case, fve Chemistry professors from Pontfcal Catholic University of Peru (PUCP) were invited to assume the role of judges in assessing the degree of coherence between the concepts and principles to be assessed with each item and the item statement.Also, they were asked to assess the degree of consistency between the learning objectves associated with each item and the item statement.
On the basis of the judges' evaluaton and their comments and suggestons, adjustments were made to the items in each test, to proceed with their pilot applicaton and a reassessment by three Chemistry professors from PUCP who partcipated as judges.

Pilot applicaton
The tests were applied with students of General Chemistry 2 course (PUCP) following a hybrid PBL approach in diferent semesters between 2010 and 2011.The ages of the students involved ranged between 17 and 20 years, with the highest percentage at the age of 17 years (43.2%).

. Informaton about partcipants in the pilot implementaton of each test
The test applicaton was always made at the end of the corresponding thematc unity.The maximum duraton of applicaton was 50 min.With the tests responses, it was performed the analysis of items for each of the three tests.For this purpose it was determined: • the percentage of omissions, corresponding to reactve responses that have been lef in blank, • the difculty degree, which is a measure of the number of subjects who responded correctly to the reagent; • biserial point coefcient (rpb), which is a special case of Pearson correlaton for determining the correlaton between the reactant and the total score and consttutes an internal discriminaton index.

Content validaton
Table 3 summarizes the results of the second judges' evaluaton of the built tests.The evaluaton by the second group of judges was satsfactory, qualifying the degree of consistency between the questons and the learning objectves, concepts and principles associated, as good or very good.The average of degree of difculty for the Thermodynamics test was 57.07%; for Kinetcs test the value obtained was 68.98% and for Chemical Equilibrium test was 51.24%.According to this, the second test can be classifed as moderately difcult, while the frst and the third tests would qualify as medium difculty.The variability of the values obtained for the items in each test was adequate, showing that it contained questons of diferent levels of difculty for students.Biserial point coefcient (rpb) is located in the range 0.418 to 0.586 for all items of the tests, so it is an indicaton that the reagents of each test discriminate between subjects achieving high score on the test and those with low scores.

Pilot applicaton
Figure 2 shows, as an example, a comparison of the results in the evaluaton of achievements in the three levels of knowledge structure of three students, using Thermodynamics test.It can be seen that the student RBS had very satsfactory results according to the learning goals of PBL methodology.The student BBG had very good levels of achievement in concepts and principles, but must improve his skills in the third level.Something similar happens with the student DRI, although his results are lower, especially in the third level, so he may require special atenton.
The overall results in the applicaton of these instruments with a group of students allow the professor to get informaton about achievement in terms of expectatons in line with PBL approach.According to it, the facilitator can have evidence to reformulate the problems, learning actvites or his facilitaton.
For example, Figure 3 shows that the results in Chemical Equilibrium test were lowest in the group of students assessed, so it would be necessary to analyze the reasons of this situaton for planning correctve acton.

CONCLUSIONS
The results obtained in the validaton of the built instruments are highly satsfactory, since, frst, ensure a high degree of consistency between the learning objectves, the content assessed and the items in each test.The theoretcal foundatons of the model have been applied rigorously and are relevant to the learning contexts promoted by PBL methodology.
In relaton to the analysis of items in each of the tests, it can be seen that a very good range is reached in the coefcient of internal discriminaton, which is very valuable for future studies using these instruments.It can be corroborated also a trend toward higher levels of difculty in items of the third level of knowledge structure.It is noteworthy that in PBL implementatons are expected to have signifcant achievements especially at this level.
As noted in earlier sectons of this report, there is a major difculty for the evaluaton of achievements in the implementaton of PBL methodology, which is related to the constructon and applicaton of appropriate tools for measurement.Sugrue's model provides a suitable and understandable framework that supports the criteria for the constructon of test to assess problem-solving skills, such as demonstrated in our investgatons.In this study, the model helped the development of three instruments to assess achievements in three basic thematc units of General Chemistry.According to the results of the validaton, the tests built have robust features for their use as tools to monitor student achievement in three levels of knowledge structure using Sugrue model, as well as in future research to explore achievements in the use of PBL methodology in contexts similar to that reported here.
common atributes and propertes, and are identfed by the same name.The evaluaton of the understanding of a concept implies: selectng examples of the concept (in a multple choice format), or generatng of examples of the concept (in an open-ended format), or explanaton of why examples refect concept atributes (in a hands-on format).

Figure 2 .
Figure 2. Evaluaton results on the three levels of knowledge structure of three students in a General Chemistry course

Figure 3 .
Figure 3. Global results on the three levels of knowledge structure for each test

Table 1
Table 1 reports further informaton about partcipants in the pilot implementaton of each test.

Table 2 .
Table 2 summarizes the fnal structure of each test.Final structure of each test

Table 3 .
Summary of second judges' evaluaton of the built tests

Table 4 .
Table4summarizes the results of analysis of items for each of the three tests.The percentage of omissions is not included, because in all cases the partcipatng students answered the total of questons.Summary of the results of analysis of items for each test