A review of project-based learning in higher education: Student outcomes and measures

0883-0355/ © 2020 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY license (http://creativecommons.org/licenses/BY/4.0/). T 1.1. Project-based learning Project-based learning (PjBL) refers to an inquiry-based instructional method that engages learners in knowledge construction by having them accomplish meaningful projects and develop real-world products (Brundiers & Wiek, 2013; Krajcik & Shin, 2014). Krajcik and Shin (2014) indicated six hallmarks of PjBL, including a driving question, the focus on learning goals, participation in educational activities, collaboration among students, the use of scaffolding technologies, and the creation of tangible artifacts. Among all these features the creation of artifacts that solve authentic problems is most crucial, which distinguishes PjBL from other studentcentered pedagogies, for example, problem-based learning (Blumenfeld et al., 1991; Helle, Tynjälä, & Olkinuora, 2006). This creation process requires learners to work together to find solutions to authentic problems in the process of knowledge integration, application, and construction. Instructors and community members (e.g. clients), normally as facilitators, provide feedback and support for learners to assist their learning process. Several review studies have predominantly focused on PjBL in post-secondary education. Helle et al. (2006) discussed both the practice of PjBL and the impact of PjBL on students’ learning. Regarding the practice, the authors found that most of the studies reviewed were confined to course descriptions in terms of course scope, instructor requirements, and team size. As for the impact, the review found that only a few studies investigated the influence of PjBL on student learning related to either cognitive (e.g. knowledge) or affective outcomes (e.g. motivation). In another study, Ralph (2015) reviewed fourteen studies that adopted PjBL in STEM education. It turned out that PjBL increased the development of both learners’ knowledge and skills. Students also felt that PjBL encouraged their collaboration and negotiations within the group. However, some students reported a lack of motivation for teamwork. Reis, Barbalho, and Zanette (2017) reviewed studies of PjBL in engineering education by adopting bibliometrics (e.g. analysis of keywords) and classifying research methods from the studies reviewed. Bibliometric results showed that, for example, the top three keywords used were project-based learning, engineering education, and problem-based learning. The classification results revealed that more than 70 % of studies focused on undergraduates and case study was the most frequently adopted research approach. In addition, some studies showed that students’ academic knowledge, skills, and motivation were improved after PjBL although students also reported difficulties of PjBL (e.g. time-consuming). However, this review had a significant limitation: the authors did not distinguish project-based learning from problem-based learning. 1.2. This study Although these reviews have mentioned student learning outcomes to a certain extent, there is no comprehensive picture of learning outcomes that can be connected to PjBL, especially in higher education. Therefore, in the current study, we will provide an overview of student outcomes of PjBL in higher education based on a review of empirical studies. To fully understand student outcomes, two research questions will be answered in this review: (1) What student outcomes of PjBL are evaluated in higher education? (2) What instruments are adopted to measure student outcomes? 2. Method 2.1. Search We used the federated search service provided by Leiden University Libraries which includes a variety of important Educational and Psychological Sciences databases, including EBSCOhost (including Academic Search Premier, APA PsycArticles, APA PsycInfo, ERIC, Psychology and Behavioral Sciences Collection), Elsevier/ScienceDirect, and Web of Science. Google Scholar and Research Gate, as external resources, were also used. Moreover, in addition to searching from the databases, we also adopted the snowballing method to identify relevant studies. The following search terms or combinations of terms and the Boolean parameters were used and presented in this way: Title contains “project-based” AND Title contains learning OR curriculum OR curricula OR course OR courses AND Any field contains “higher education" OR undergraduate OR graduate OR “post-secondary” OR tertiary AND Any field contains outcome OR impact OR influence OR effectiveness. The publication date of the articles was before September 2019. The material type of the results was Articles, and the language of these studies was English. In addition, all the articles were confined to peer-reviewed articles. In total, 450 articles were found. 2.2. Selection Articles were further selected manually. The following selection criteria were applied: (a) the studies had to be empirical and should provide original data; (b) the studies had to focus on student learning; (c) the process of PjBL had to be conducted in higher education; (d) the impact of PjBL on student learning outcomes (i.e. cognitive, affective, and behavioral outcomes) had to be measured; (e) the studies had to meet the key characteristic of PjBL, namely the report of the creation of artifacts. Therefore the following types of studies were excluded: non-empirical studies and meta-analyses, studies which did not distinguish project-based learning from problem-based learning, studies that did not focus on student learning, studies conducted in non-tertiary contexts, P. Guo, et al. International Journal of Educational Research 102 (2020) 101586


Introduction
In recent years institutions of higher education have been trying to provide students with both hard skills, namely cognitive knowledge and professional skills (Vogler et al., 2018), and soft skills, such as problem-solving and teamwork (Casner-Lotto & Barrington, 2006). However, these skill related goals are not easy to be achieved as traditional learning has been playing a prevailing role where teachers are "the transmitter of the knowledge" while students act as "the receptor of the information" (Alorda, Suenaga, & Pons, 2011, p. 1876. As a result, it is difficult for students to fully engage in educational practices, which may lead to a superficial understanding of disciplinary knowledge. Besides, universities, and research universities, in particular, are more focused on the cultivation of students' research skills rather than professional skills or transferable skills. Thus, this might cause a gap between what students learn at the university and what they need in the workplace (as cited in Holmes, 2012). In order to change this situation, it is suggested that students are provided with the opportunity to participate in real problem-solving and knowledge construction in authentic professional contexts. One attractive way to achieve this goal is through project-based learning (PjBL). In Chen & Yang's (2019) review, the effects of PjBL and teachers' direct instruction on students' academic achievement in primary, secondary, and tertiary education were compared. PjBL in this study indicates a learning process in which students are engaged in working on authentic projects and the development of products. The results demonstrated that PjBL had a more positive impact on students' academic achievement than direct instruction did. However, it turned out that only 20 % (6 out of 30) studies reviewed were conducted in higher education. In addition, Lee, Blackwell, Drake, and Moran (2014) claimed that -compared to the progressive development of PjBL in K-12 education-the investigation of PjBL in higher education has been left behind. Therefore, the current study aims to contribute to a better understanding of PjBL implemented in higher education.
https://doi.org/10.1016/j.ijer.2020.101586 studies focusing on the development of PjBL curricula/activities/technologies and on the implementation/practices of PjBL, studies that measured the influence of tools/frameworks on PjBL, and studies that lacked clear reports of artifacts.
Ten percent of 200 articles were rated by a co-author via the selection criteria mentioned above. The result showed that there was a 100 % match between the two raters. Ultimately, a total of 76 articles were selected for review.

Analyses
Based on the content of the selected articles, we have set up a matrix that involved the research design, learning outcomes, measurement instruments, findings, and limitations of the studies reviewed. Based on this matrix, we summarized the outcomes that were measured and the instruments that were used to measure these outcomes based on commonly used clustering of learning outcomes and research methods (as used in Brinson, 2015 andPost, Guo, Saab, &Admiraal, 2019). We divided the outcomes into four categories, namely cognitive, affective, behavioral outcomes, and artifact performance. Five categories of instruments were revealed, including questionnaires, rubrics and taxonomies, interviews, tests, and self-reflection journals.

Results
As can be seen in Table A1 (Appendix A), more than half of the studies reviewed (n = 54) involved only one group. Moreover, both self-reported and externally measured learning outcomes and measurement instruments were reported in the 76 studies reviewed. We will present the findings for each learning outcome and for each type of learning outcome we will present instruments that are used to measure these learning outcomes.

Knowledge
In 17 studies, students' content knowledge, conceptual understanding, and course achievement were reported as outcomes of PjBL. For example, biological knowledge, such as cloning and DNA isolation (Regassa & Morrison-Shetlar, 2009), psychological knowledge relevant to healthy living habits and pressure management (Lucas & Goodman, 2015), and technical knowledge related to space engineering (Rodríguez et al., 2015), were investigated. Students' academic performance of programming course was measured in Çelik, Ertaş, and İlhan, (2018).
Four types of instruments (i.e. self-report questionnaires, tests, rubrics, and artifacts) were adopted to measure students' knowledge, in which self-reported questionnaires were most applied. Both Likert scales (e.g. Lucas & Goodman, 2015;Rodríguez et al., 2015;Torres, Sriraman, & Ortiz, 2019) and qualitative questionnaires with open-ended questions (e.g. García, 2016;Luo & Wu, 2015) were adopted. For example, Katsanos, Tselios, Tsakoumis, and Avouris (2012) required students to evaluate their knowledge of web accessibility on a Likert scale from 1 (very low) to 5 (very high). Tests were the second frequently used tools to assess students' academic knowledge (e.g. Çelik et al., 2018;Katsanos et al., 2012;Mohamadi, 2018). For example, students' self-directed knowledge was measured by written tests with knowledge-based, application-based, analysis-based, and synthesis-based questions (Chua, 2014;Chua, Yang, & Leo, 2014). In Regassa and Morrison-Shetlar (2009), concepts of biology were examined with a test with three multiple-choice and seven open questions. Only one study (i.e. Kettanun, 2015) measured students' course performance with rubrics. In this study, English learners' presentation was evaluated via six criteria, such as how authentic the words they used and how well they organized the facts and opinions. In another study, Barak and Dori (2005) evaluated students' understanding of chemistry via the analysis of their projects.

Cognitive strategies
Nine studies measured the cognitive learning strategies that students adopted in PjBL. For instance, students in Wu, Hou, Hwang, and Liu (2013) adopted seven strategies, including remembering, understanding, applying, analyzing, evaluating, creating, and straying off-topic. Similarly, learners in Stozhko, Bortnik, Mironova, Tchernysheva, and Podshivalova (2015) also used seven strategies, which were divided into four levels, namely lower level (identification), basic level (knowledge and comprehension), middle level (application and analysis), and upper level (synthesis and evaluation). Both Heo, Lim, and Kim (2010) and Hou, Chang, and Sung (2007) identified students' five phases of knowledge construction, namely information sharing, disagreement detection, negotiation of meaning, modification of new ideas, and agreement statement. In the study of Helle, Tynjälä, Olkinuora, and Lonka (2007), two cognitive processing strategies of students were investigated, namely relating (i.e. the connection of new knowledge to prior information) and structuring (i.e. the outline of a set of ideas).
Five types of instruments (i.e. rubrics/taxonomies, questionnaires, interviews, observation, and artifacts) were used to assess students' learning strategies, in which rubrics and taxonomies were most frequently adopted (e.g. Hou et al., 2007;Usher & Barak, 2018). For example, Heo et al. (2010) developed and used a grading rubric with several criteria, such as learners' understanding of the design value and their creativity. Both Stozhko et al. (2015) and Wu et al. (2013) adopted the revised Bloom's Taxonomy to assess students' cognitive strategies. However, they used different operationalization of this taxonomy. Other studies used questionnaires as the assessment tools (e.g. Biasutti & EL-Deghaidy, 2015). Stefanou, Stolk, Prince, Chen, and Lord (2013) adopted a 7-point Likert scale, with statements indicating 1 (not at all true of me) to 7 (very true of me), to assess students' learning strategies. Nine subscales, such as the strategies of organization and self-regulation, were included. Helle et al. (2007) adopted both 5-point Likert scales and semi-structured interviews to investigate students' cognitive processing. Barak and Dori (2005) determined students' four levels of chemistry understanding by the analysis of students' projects, classroom observation, and student interviews.

Affective outcomes
The affective outcomes are distinguished into both evaluations by students about what they learned (i.e. whether PjBL was effective) as well as how they perceived the learning experience.

Skills
Nine studies explored both students' hard skills and soft skills in PjBL. Hard skills, such as marketing skills for students of hotel administration (Vogler et al., 2018), general care skills for nursing students (Wu et al., 2018), EFL learners' writing skills (Sadeghi, Biniaz, & Soleimani, 2016), and the skills of students of engineering management to decide where to locate public services in real-life situations (Berbegal-Mirabent et al., 2017), were reported. Besides hard skills, several soft skills were reported, such as skills of problem-solving and critical thinking (Vogler et al., 2018;Wu et al., 2018;Wurdinger & Qureshi, 2015), collaboration and team working skills (Berbegal-Mirabent et al., 2017, p.;Rodríguez et al., 2015;Vogler et al., 2018), and lifelong learning skills (Vogler et al., 2018;Wu et al., 2018). For example, Brassler and Dettmers (2017) emphasized student problem-solving skills from three interdisciplinary perspectives: (a) considering and applying different views, (b) re-considering the strategies used, and (c) adopting P. Guo, et al. International Journal of Educational Research 102 (2020) 101586 discipline-based methods. Some phases to solve a scenario-based problem, such as problem identification, data collection and analysis, and back-up plan design, were investigated in Chua (2014) and . Five types of instruments (i.e. questionnaires, tests, rubrics, interviews, and reflective journals) were adopted to assess students' skills, in which questionnaires were most adopted (e.g. Rodríguez et al., 2015;Wu et al., 2018;Wurdinger & Qureshi, 2015). For example, Brassler and Dettmers (2017) used a self-reported scale which was adapted from previous research. Several development steps, including literature review, concept identification, focus group interview, items creation, pilot study, and revision, were used to revise the scale. Scenario-based tests were developed by instructors and used in Chua (2014) and . In these studies, students' performance in applying strategies to solve problems related to industrial drying was assessed with tests. A rubric for assessing students' technical skills through oral presentations was adopted in Berbegal-Mirabent et al. (2017). Students' performance was evaluated by the content, comprehension, and style of the presentation and ranked in four levels (from advanced to inadequate). Also, Vogler et al. (2018) adopted both self-reflection journals and focus group interviews to assess learners' skills.

Engagement
Four studies focused on students' learning process in PjBL. Learners' perceived engagement was reported in Cudney and Kanigolla (2014). Three aspects of students' engagement, i.e. the level of general involvement in the semester project, the degree of participation in class discussions, and whether they applied the course concepts to practice were investigated. In Fujimura (2016), the educational activities that students participated in during the whole project, such as making a research plan and collecting and analyzing the data, were explored. Moreover, the process of how students learned content knowledge was also examined. In Hou (2010), learners' seven behavioral patterns, including project topic analysis, data collection, data evaluation, project content analysis, comprehensive analysis, comments proposal, and irrelevant information discussion were explored. In Koh, Herring, and Hew (2010) five levels of student knowledge construction, namely sharing, trigger, exploration, integration, and resolution, were examined in both PjBL and non-PjBL activities.
A five-point Likert scale (from strongly agree to strongly disagree) with 23 questions was adapted from Yadav, Shaver, and Meckl (2010) and used to assess students' level of involvement in the semester project (Cudney & Kanigolla, 2014). Students' online discourse was recorded to get insight into their learning process in Hou (2010) and Koh et al. (2010). In Fujimura (2016), both student reflection journals and audio-recordings of discussions were used to determine their learning activities. Apart from these two instruments, three more instruments, namely the artifacts created by students, students' reflection journals, and focus group interviews with students, were also adopted to investigate student learning process.

Artifact performance
Three types of artifacts (see Table A1), i.e. physical objects, documents, and multimedia were most frequently measured in ten studies reviewed. All products were assessed by rubrics. For example, Chua (2014) and  assessed the dryers that students created by a 5-point rubric made by instructors. The grading criteria included, for example, original design and product quality. Papastergiou (2005) evaluated the website that students created by five criteria, including topic, content and aesthetics, pedagogy, technology, and usability. Rajan, Gopanna, and Thomas (2019) assessed students' project reports by a 5-point rubric (from excellent to poor) for several writing tasks, such as literature review, analysis, and presentation. Torres et al. (2019) evaluated students' bid reports based on three aspects, including accuracy of report (40 %), completeness of report (40 %), and neatness of report (20 %).

Discussion
Learners' knowledge, strategies, and skills were frequently measured by most instruments, namely self-reported questionnaires, rubrics, tests, interviews, observation, self-reflection journals, and artifacts. These learning outcomes received much attention might because employers report that basic knowledge and skills are essential for students' readiness to work (Casner-Lotto & Barrington, 2006). Students' perceived benefits and experience of PjBL were measured by questionnaires, interviews, observation, and selfreflection journals. However, although these two outcomes were distinguished from each other in this review, in many studies reviewed they were intertwined, which causes difficulties to interpret the findings. Student engagement was evaluated by questionnaires, interviews, self-reflection journals, artifacts, and recordings of students' discussions in only four studies. It is necessary to investigate the specific learning process of students in future studies. All artifacts were assessed by rubrics. However, the evaluation of products has not received much attention in the studies analyzed although it is the product creation that differentiates PjBL from other forms of learning. The creation of products is of importance because it helps learners to integrate and reconstruct their knowledge, discover and improve their professional skills, and increase their interest in the discipline and the ability to work with others. In other words, the final products are the concentrated expression of various competencies that students may develop during PjBL. Thus, future studies are suggested to investigate more about the performance of students' final products.
Many studies reviewed lacked clear descriptions of measurement instruments and data analysis. Although questionnaires were most frequently used, some studies did not report the items of the questionnaire (e.g. Balve & Albert, 2015;Costa-Silva et al., 2018;Davenport, 2000;Hogue, Kapralos, & Desjardins, 2011;Ngai, 2007;Seo, Templeton, & Pellegrino, 2008). There was also a lack of clear reports of the reliability and validity of scales (e.g. Dehdashti et al., 2013;Sababha, Alqudah, Abualbasal, & AlQaralleh, 2016;Thomas & MacGregor, 2005;Yam & Rossini, 2010). These limitations were also found in self-reported questionnaires used in other studies like clinical research (Kosowski et al., 2009). Providing information about the psychometric properties of instruments benefits researchers' selection of high-quality tools and the results of their studies (C. de Souza, Alexandre, & de B. Guirardello, 2017). Future research should be improved by reporting the items, reliability, and validity of the instruments adopted. As for the analysis of qualitative data, several studies (e.g. Kettanun, 2015;Regassa & Morrison-Shetlar, 2009;Zhang et al., 2009) lacked quality checks. Standardized audit procedures (e.g. the method introduced in Akkerman, Admiraal, Brekelmans, & Oost, 2008) are recommended to adopt to ensure the quality of future studies.
In addition, since computer technologies are frequently used in PjBL, the use of log data, as a data collection method (e.g. Lewis, Easterday, Harburg, Gerber, & Riesbeck, 2018), should be further considered. A more comprehensive image of student learning can be provided by log data (Deane, Podd, & Henderson, 1998) based on a variety of behavior, such as browsing content, times, frequency, that are recorded. Moreover, log files are suitable for discovering and analyzing students' learning strategies and patterns in a complicated cognitive learning process like complex problem solving (Greiff, Niepel, Scherer, & Martin, 2016). Thus, this additional information helps teachers and researchers understand more about student profiles (e.g. student interest and engagement) and improve curricula in the future (Bunderson, Inouye, & Olsen, 1988).
Although this study did not intend to focus on the impact of PjBL on student learning, a small number of studies reviewed have proved that PjBL benefits students' content knowledge (e.g. Alsamani & Daif-Allah, 2016; Mohamadi, 2018), learning strategies (e.g. Barak & Dori, 2005;Stefanou et al., 2013), skills (e.g. Brassler & Dettmers, 2017;Wu et al., 2018), motivation (e.g. Helle et al., 2007;Wu et al., 2018), and product quality (e.g. Affandi & Sukyadi, 2016;Torres et al., 2019). However, it is difficult to determine the effects of PjBL on student learning as most of the studies analyzed did not implement research designs that allow claims about effects on learning outcomes. Therefore, for future research, we recommend that more experimental research should be done to determine the benefits of PjBL on students' diverse learning outcomes.

Implications
Since project-based learning and problem-based learning are similar and there is still debate about their effects on student learning, we need to differentiate between them, especially in higher education. A crucial task of higher education is to provide innovative education for students who enter the labor market in the future as it raises their competitiveness and promotes the development of the society in the long term (Crosling, Nair, & Vaithilingam, 2015). Research has suggested fostering students' innovation by supporting their autonomy during learning tasks (Martín, Potočnik, & Fras, 2017). Project-based learning can meet such needs. Although several studies (e.g. Braßler, 2016;Helle et al., 2006) have indicated differences between project-and problembased learning, such as different types of tasks and role of the instructor, however, the way of processing knowledge is the key. The focus of problem-based learning lies in knowledge application while project-based learning, which is based on the learning science of active construction (Krajcik & Shin, 2014), emphasizes knowledge construction. This process of creating new knowledge allows students to test and achieve their ideas in the way they want, which promotes their innovation competence. Thus, we believe it is necessary to encourage teachers in higher education to adopt project-based learning. Besides, although disciplines were not analyzed in this review, there are many applications of project-based learning in STEM education. Future research should consider implementing project-based learning more in the field of humanities and social sciences.

Concluding remarks
To conclude, this review has found four categories/seven sub-categories of student learning outcomes in PjBL in higher education and eight corresponding measurement instruments. More studies should be conducted to evaluate student learning processes and the performance of students' artifacts. The quality of measurement instruments should be reported and the way of data analysis should be enhanced. Besides, more experimental research should be conducted to determine the effects of PjBL on student learning.

Declaration of interest
None.