THE EFFECTIVENESS OF PBL PROBLEMS FROM STUDENTS AND TUTORS PERSPECTIVES

Background: PBL problems are used as a starting point to students in attaining learning objectives. Students are expected to discuss the problems according the principles of collaborative learning, constructive learning, contextual learning, and self-directed learning. A continuous monitoring, evaluating, and improving the PBL problems should be done systematically. Nevertheless, the problems are lacking monitoring and evaluation. The aim of this study is to evaluate the effectiveness of PBL problem by providing an evaluation tool. The evaluation involved students and tutors. Method: A survey by using questionnaire based on six factors of an effective PBL problem was conducted to obtain students and tutors perception towards the effectiveness of 6 PBL problems used in tutorial session. The questionnaire for each scenario distributed to four randomly selected group tutorials (total samples per scenario is 40 students and four tutors). The perceptions were then analysed by using descriptive statistic. Four open ended questions included in the questionnaire were used to get better understanding and explanation of quantitative result. Results: A total of 198 (82.5% response rate) students and 23 (95% response rate) tutors filled the questionnaire. Students and tutors gave their score for the overall quality of each PBL problems. Some differences among students and tutors in rating the overall quality on each problem were found. The strength and the weakness of each problem were also obtained from students’ and tutors’ rate for the six factors in the questionnaire. Each group of participants had their own concern regarding the most important factor for an effective PBL problem. Conclusion: The result of this evaluation could reflect the effectiveness of PBL problems in achieving students’ learning objectives from different viewpoints: students and tutors. This valuable information can be used by problem designers and their institutions to monitor and improve the quality of PBL problems continuously.


INTRODUCTION
One of the influencing factors to make problembased learning (PBL) successful is quality of problems. A PBL problem should lead students to have meaningful discussion based on the principles of constructive learning, collaborative learning, contextual learning and self-directed learning. 1 As the core of problem-based learning, it will lead students to actively discover what they should learn based on the provided problems. 2,3 It can be explained that creating an effective PBL problem needs a careful consideration in determining the intended learning objectives which should be appropriate with students' prior knowledge. The integration between new information with existing knowledge will generate new ideas that promote the construction of knowledge. A complex and attractive problem would stimulate students in thinking, analysing and reasoning. It is expected that students will interact among each other in a collaborative way which would maximize their learning process. Relevant context of the problem and its clear linkage to students' future profession are also important to stimulate transfer of knowledge and to enhance students' interest in subject matter. The principles described would motivate both internal and external motivation of students to engage in self-study. 4,5 Steps in constructing a PBL problem Several literature sources have been written in which some considerations to construct an effective PBL problem are described. Dolmans & Snellen-Balendong 1 explain about the importance of educational objectives related to type and format of problem. Other articles describe several conditions that should be met by a problem. 6,7 Schmidt & Moust 6 also distinguish four type of problems based on different categories of knowledge: explanatory knowledge, descriptive knowledge, procedural knowledge and personal-normative knowledge. From the literatures cited above, we could define several main steps to construct an effective PBL problem.
First, student's learning objectives must be clearly defined, relevant with student's prior knowledge and coherent with the institution's learning objectives. This is important to promote integration of knowledge that reflects constructive learning process. 1,6,7 Second, choose the right type of problem which is appropriate for the providing learning objectives. Four types of problems have been distinguished based on four types of knowledge that would be obtained by students 6 : Explanation problem, fact-finding problem, strategy problem, and moral-dilemma resolution problem. boredom. Some examples are using virtual tutorial, soft ware, and e-learning. It is really expected that by using various problem format with various discussion style will make the learning process becomes more efficient and effective. 8 Fourth, a problem designer has to considering several aspects that should be met by a problem. The problem designer should provide a problem that is realistic and contextual according to the student's environment and future profession. The problem should also be formulated using appropriate cues, terms and language which are interesting to stimulate student's prior knowledge. This might also bring impact to stimulate discussion and motivate students to learn and to look for relevant literatures. 4,6,7 Fifth, the problem should be reviewed by a team with proficiency in PBL before being used by students. It is necessary since each problem designer has different ability in constructing a well-structured problem. 9 Sixth, to ensure that the problems have met the criteria of effective problem, a systematic and continuous evaluation should be undertaken. 1 It can be done by students and tutors at the end of the discussion. 9 The evaluation result will provide teachers, planning groups and stake holders with useful information to monitor and improve the quality of problems.

A tool to evaluate the effectiveness of PBL problem
A study conducted by Munshi, Zayat and Dolmans 4 has result a tested questionnaire to evaluate PBL problems. The questionnaire had been elaborated from several literature sources which describe criteria of effective PBL problems. The capability to assess the strength and the weakness of PBL problems also had been proven. This questionnaire consists of six factors that describe criteria of an effective PBL problems: (1) stimulates thinking, analysing and reasoning, (2) stimulates self-directed learning, (3) leads to studying the intended contents, (4) enhances the interest of subject matter, (5) relevance to the future profession with realistic context and (6) matches the level of prior knowledge. Each factor has three detailed items. One additional item for the overall rating of the quality of a single PBL problem was included. This questionnaire also includes four open ended questions concerning the strength and weakness of PBL problem, suggestions for improvement, and further comments.
The aim of this study is to evaluate the effectiveness of PBL problem by providing an evaluation tool. The evaluation involves students and tutors to fill out the PBL problem questionnaire by Munshi, Zayat & Dolmans. The reason in involving students and tutors to evaluate the quality of PBL problem is that they use the problem directly in the tutorial process. Dolmans & Snellen-Balendong 1 stated that students and tutors have crucial part in the PBL learning process. Active involvement of students and encouragement from tutor in stimulating student's learning were affected by the quality of PBL problem. Thus, students and tutors are expected to provide proper information related to the quality of the problems.
This study is focused on comparing perception of students and tutors in valuing PBL problems. Therefore, the effectiveness of a sample of PBL problems are estimated by using 6 factors described on PBL problems questionnaire by Munshi, Zayat and Dolmans. The research questions of this study are: (1) What is students' perception of the quality of PBL problems used in tutorial session? (2) What is tutors' perception of the quality of PBL problems used in tutorial session? (3) Is there any different perception between students and tutors regarding the quality of PBL problems used in tutorial session?

METHOD
A survey by using questionnaire is conducted to obtain students and tutors perception about particular PBL problems used in tutorial session. Quantitative study was conducted to obtain students and tutors perception about the effectiveness of PBL problems by giving score to the items in the questionnaire. Four open ended questions included in the questionnaire were used to get better understanding and explanation of quantitative result.

Setting
This study is conducted in the undergraduate medical curriculum at Faculty of Medicine Universitas Gadjah Mada in the academic year 2011. It had been submitted to ethical approval committee of FMUGM. This school has 7 semesters competencebased curriculum for undergraduate program which use PBL as instructional design. Each semester has 3 blocks. Each block consists of 6 tutorial sessions. Each tutorial consists of two-hours analysis and two hours discussion within a week to discuss a problem. Each group consists of 8 to 10 students and facilitated by one tutor. The questionnaire was administered to the third block in year 1, 2 and 3 with a total of 6 problems. Problem 1 and 2 were used by first year students, problem 3 and 4 were used by second year students while problem 5 and 6 were used by third year students.

Subjects
The population of this study is all undergraduate medical students of FM UGM from year 1, 2, and 3 (N = 1200). Students' age ranged from 16 to 18 years-old for year 1, 17 to 20 years-old for year 2 and 18 to 21 years-old for year 3. Tutors population are teachers of FM UGM who were being tutors for undergraduate medical students in year 1, 2, and 3 (N = 120). Age ranged from 24 years old to 64 years old. All tutors had undergone basic training for tutorial. We use PBL problems from the third block of three consecutive years. Two problems had been chosen randomly for each block. A total of 6 problems were evaluated by students and tutors. The sampling method was figured as follows: The sample of this study was from all of tutorial groups. Each block had a range of 39-43 tutorial groups with different tutors in each group and an average of 8-10 students. For each problem, the tutors and students from 4 tutorial groups were asked to fill out the questionnaire to rate a problem. The tutors and students were given the option to keep their identities anonymously. The tutors' total response rate was 95% (n = 24), while the students' total response rate was 82.5% (n = 240).

Instrument and data collection
A questionnaire from Munshi, Zayat and Dolmans 4 was used in this study. The questionnaire had not been statistically validated, however it was constructed based on review of literature about criteria for qualified PBL problems and it had been accepted that the questionnaire had been validated from literature. The questionnaire consists of six factors that describe an effective PBL problem: The questionnaire covered all the items of six factors and one additional item for providing an overall score on the problem in general. Four open ended questions were also provided to give opportunity for participants to describe the problem's strengths and weaknesses, suggestions for improvement and further comments.
The questionnaire was provided in English based on consideration that some of terms in the questionnaire would be difficult to be translated into Bahasa Indonesia. All students in FMUGM have minimum TOEFL score of 450 and all block books were written in English, so that it could be expected that students will find minimum difficulty to understand the questionnaire. Eight teaching staffs were willing to be a research assistant to accompany students and tutors to fill out questionnaires. The purpose was to help only with clarifying any item in the questionnaire if necessary. They received a short instruction to prepare them on their role.
In this study, the following general procedures for data collection were conducted. The researcher and research assistants explained to the students and tutors carefully about how to fill the questionnaire. Researcher and research assistants then stayed with participants in the room while they filled the questionnaire. Researcher and research assistants would give clarification when was needed by the participants. The questionnaire was administered to the third blocks in year 1, 2, and 3. It was filled by students and tutors right after the second tutorial meeting of a selected problem.

Data Analysis
Descriptive statistics of overall rating item (item 19) of the six problems were computed for students and tutors. Descriptive statistics were also computed focusing on each of the six factors. Score for each factor was the average of the scores of the three items. The standard to interpret the average score of all factors and overall rating item was: l 3 or less is insufficient, l More than 3 to 3.5 means that improvement is needed, l More than 3.5 to 4 is sufficient l More than 4 is good (scale 1-5).
While the overall item score was on a scale of 1-5, 1 was scored as insufficient, 2 was scored as reasonable, 3 was scored as sufficient, 4 was scored as good and 5 were scored as excellent. 4 The qualitative data were obtained from written feedback for strength, weakness, suggestion for improvement and further comments in the questionnaire (four open ended questions). Each feedback statement was classified based on six factors in the questionnaire. In this study two coders were used who worked individually to classify the feedback statements. For reliability purpose, the coders then conducted three meetings to check consistency and achieve consensus about differing interpretation. 9

RESULTS AND DISCUSSION
Students' perception of the overall quality of PBL problems Each problem was rated by 4 groups of students to obtain their perception about the effectiveness of a certain problem. Some students did not give their score on item 19. From table 1, it shows that students highly agreed in rating problem 3 as sufficient (a mean score above 3.5). Problem 1, 2, and 5 were also rated as sufficient while problem 4 and 6 need improvement (a mean score below 3.5). From table 1 and 2 it can be shown that students had biggest range in rating problem 5. It was more likely that students had different perception about it. However, most students were give positive impression for problem 5, and only 6 to 7 students who did not satisfy with this problem.  Overall, students tend to satisfy with the quality of all PBL problems. The highest scoring problem rated by students had strength on its relevance between the problems to the future professions with realistic context (factor 5). A contextual problem that was sufficiently presented could be expected to motivate students' learning. This also will influence the students' amount of time on self-study. 6,11 "This problem relates with everyday problem, it stimulates the prior knowledge and eagerness to find the cause of the problem" -student "This problem was related to a clinical problem; so that it is strongly stimulate an integrated discussion" -student The weakness of the lowest scoring problem rated by students was concerned on the ability to stimulate thinking, analysing, and reasoning (factor 1). Minimal direction to lead students to discuss a particular topic or field would be an obstacle in achieving learning objectives. 6,11 "The clues are less, i.e. risk factors and general information about the patient" -student

Tutors' perception of PBL problems quality: overall view
Each problem was rated by 4 tutors. Some tutors did not give their score on item 19. From table 3 and 4, it can be shown that problem 6 rated as good (a mean score above 4). Problem 2 and 5 were rated as sufficient (a mean score above 3.5). The mean score of problem 3 and 4 were not so high, but from table 2 it can be assumed that the problems had been sufficient. Problem 1 had the biggest score range. One tutor seemed not satisfy with problem 1, while the other two tutors were satisfied.    Overall, tutors tend to satisfy with the effectiveness of all PBL problems. The highest scoring problem rated by tutors had strength on the ability in stimulating thinking, analysing and reasoning (factor 1); and the relevance between the problem and the future profession in realistic context (factor 5). On the other hand, the weakness of the lowest scoring problem rated by tutors was also concerned on the ability in stimulating analysing, thinking, and reasoning.
"This problem lead to discuss about awareness, early treatment and early detection to vascular disease"the strength, tutor "This scenario is too focusing on milk, diarrhea and mal absorption while the intended learning objectives are still in basic concepts. The effect is that students tend to see and discuss the pathological condition"-the weakness, tutor. A tutor gave a comment regarding the case which underlies the problem.
"This scenario used a rare case, the discussions spend much time in clinical aspect only" -tutor Presenting a relevant and realistic context is an essential point in constructing a problem and it also should be adapted with students' prior knowledge. 1,6,7,11 The inappropriateness between the intended competencies in students' future professions and the underlying case might decrease the contextualisation of knowledge. Table 6 is comparing between students 'and tutors' perception of the overall quality of each PBL problem. It can be described that problems 2, 4 and 5 were scored similar by students and tutors. On the other hand, each group of participants had different perception from problems 1, 3, and 6. For example in problem 1, students agreed that it was sufficient, but tutors did not think so. The interesting finding was also found in problem 6 which showed different perception between two groups of participants. Students thought that improvement was needed for problem 6; while tutors highly agreed that it was a good problem.

Students and, tutors' perceptions of each PBL problems.
From the description above, it might be interesting to look into detail about the strengths and weaknesses of the problems based on six factors in the questionnaire. Table 7 and 8 show students' and tutors' perceptions based on six factors of each PBL problem.

Factor 1: Stimulates thinking, analysing and reasoning
Highest score for factor 1 was rated by students on problem 4 (mean score = 3.54) and by tutors on problem 2, 3, 4, 5 and 6 (mean score ranged from 3.83 to 4.33). The tutors rated the lowest score on problem 1 (mean score = 3.5); while students give the lowest score for factor 1 on problem 6 (students' mean score =3.38) According to students, problem 4 had the best ability to stimulate thinking, analysing and reasoning. On the other hand, problem 6 had the worst ability to stimulate thinking, analysing and reasoning according to students. In a contrary, tutors gave a high score for problem 6.

Factor 2: Stimulates self-directed learning
Highest score for factor 2 was rated by tutors on problem 1 (mean score = 3.92). There was no highest score for factor 2 that was rated by students. Both students and tutors did not give low score for this factor.
Tutors seemed satisfied with the ability of each problem in stimulating self directed learning, and the best one was being problem 1. It was striking that students did not give the highest score for factor 2 compared with other factors. However, they seemed satisfied with the ability of each problem in stimulating self-directed learning (mean score ranged from 3.52 to 3.77)

Factor 3: Leads to studying the intended contents
There was no highest score for factor 3 that was rated by both students and tutors. Lowest score for factor 3 was rated by tutors on problem 2 (mean score = 3. There was no lowest score for factor 3 that was rated by students.
Students seemed satisfy with the ability of most problems to lead studying the intended contents although there was one problem that gets a low score, and it was being problem 4. Tutors were also likely satisfied with the ability of this factor on most problems, except problem 2.

Factor 4: Enhances interest in subject matter
Highest score for factor 4 was rated by tutors on problem 5 (mean score =4.08). There was no highest score for factor 4 that was rated by students. Lowest score for factor 4 was rated by students on problem 4 (mean score = 3.32) and problem 5 (mean score = 3.43). There was no lowest score for factor 4 that was rated by tutors.
Students seemed agree that most problems were attractively presented and it enhance their interest in subject matter. However, there were two problems that get a low score: problem 4 and 5. Like students, tutors were also satisfied with the problems' presentation, except problem 2 that seemed need to be improved.

Factor 5: Relevance to the future profession with realistic context
Highest score for factor 5 was rated by students on problem 1, 2, 3, 5 and 6 (mean score ranged from 3.70 to 4.08); by tutors on problem 4 (mean score = 4) and problem 6 (mean score = 4.33). There was no low score for factor 5 rated by students and tutors. So, the two groups of participants tend to agree that each problem was relevant to the future profession with realistic context.

Factor 6: Matches the level of prior knowledge
There was no high score for factor 6 rated by students and tutors. Lowest score for factor 6 was rated by tutors on problem 4 (mean score = 3.50); while students did not rate the factor 6 as low score.
There are both similar and different perceptions between the groups of participants in rating the problems. The similarity and differences were not only on the overall quality of the problem, but also on determining the strength and the weakness of each problem based on the six factors in the questionnaire.
From students' point of view, they tend to prefer to a problem that gives relevant professional context. Clear linkage between the provided problems with students' future profession will enhance students' interest in subject matter, thus, it will ease transfer of knowledge process. 4,5 Students also consider about how the problem stimulates them in thinking analysing and reasoning as well as tutors that seem to give concern on this factor. The problem's formulation including the appropriate cues, terms and language could gain students' interest and stimulate student's prior knowledge. This might also bring impact to stimulate discussion and motivate students to learn and to look for relevant literatures. 4,6,7 Tutors have crucial tasks in stimulating and encouraging students to have interaction one to another, to elaborate students' knowledge and to explore their reasoning skills. 7 Therefore, appropriate cues and optimal directions within a problem would help a tutor to do his/her task in facilitating students' learning process.
Both students and tutors likely agreed that most problems had been matched with students' level of prior knowledge although problem 4 seemed need to be improved.

Limitation of the study
There are some limitations of this study that need to be mentioned. First, the questionnaire has not been statistically validated, however it was constructed based on review of literature about criteria for qualified PBL problems and it had been accepted that the questionnaire had been validated from literature. Second, only a limited number of problems were analyzed in this study. To make it more representatives and have various structures, two problems from each block in three consecutive years had been chosen. The problems were developed by different block coordinators. Third, there were difference in group size and also differences within homogeneity of the group e.g. group of tutors have a big range of age. However, each tutor has same requirements to be assigned as a tutor: having certificate from basic training for tutor and following the general procedure of block implementation discussed in the introduction section.

CONCLUSION
This study provides features about how students and tutors value PBL problems as part of evaluation about the effectiveness of the PBL problems. The result of this evaluation could reflect the effectiveness of PBL problems in achieving students' learning objectives from different viewpoints: students and tutors. This valuable information can be used to monitor and improve the quality of PBL problems continuously. Further research could be done by extending the range of study i.e. sample size, among institutions with similar background. Another research could be on the utility of various types and formats of PBL problem to sustain group discussion function.
This study could contribute in PBL teaching and learning process. Evaluate the PBL problems by using different perspectives might be useful for problem designer and institution to improve the quality of problems within the PBL as trigger for achieving learning objectives in the tutorial sessions. Based on this study, a systematic process of continuous monitoring, evaluation and improvement towards the effectiveness of PBL problems could be illustrated on figure 1. Problem designer construct a problem by using a clear framework (e.g. the six steps in constructing a PBL problem). After that, the problem should be reviewed by a team with proficiency in PBL, at least 4 weeks prior to the block starting date. It is necessary, so that the problem designer still have enough time to make some revision or improvement if it is needed (redesign), according to reviewer team's suggestions and recommendations. The reviewer team might use a framework that could be obtained by doing some modification from the questionnaire for evaluation. Next, it would be better if whenever the problem is implemented, one or two representatives of the reviewer team are assigned to be a tutor. Then, after the problem has been used; students, tutors and expert tutors evaluate the effectiveness of the problem using a tool e.g. questionnaire from Munshi, Zayat, & Dolmans. 4 The evaluation results are reported to the problem designer to have further improvement.