Real-time Impact Evaluation of a Capacity-Building Health Project in Lao PDR

This study presents a real-time impact evaluation of a human capacitybuilding health project in Laos, financed by a Korean aid agency and executed jointly by Laotian and Korean higher educational agencies. The project aims to improve the health status of Laotians by enhancing practicing doctors’ clinical performance capacity, to be attained by advancing academic achievement at the University of Health Sciences (UHS) in Laos. Therefore, this real-time impact evaluation adopted the difference-in-differences regression analysis method, showing that the project improved the academic achievement of the UHS students who were taught by the project fellowship awardees more, compared to the UHS students who were taught by non-fellowship faculty members. It remains to be evaluated whether these UHS students taught by the project fellowship recipients would also perform better clinically in public hospitals in the future.

UHS, the sole higher educational institution dedicated to training medical professionals in Lao PDR, via overseas training fellowships. The aim was to improve the clinical performance of doctors at public hospitals and ultimately to achieve better health outcomes for Laotians. The project was inspired by a successful USAID-financed venture that offered SNUCM faculty members training fellowships at the University of Minnesota during Korea's development era . About 70% (77 professors) of the total faculty of SNUCM were retrained at the University of Minnesota through the USAID-financed fellowship. As a result, they not only introduced new trends in medical education methodology and organizational culture to SNUCM, but they also diffused them to other universities in Korea, contributing to improved clinical performance at hospitals across Korea (Ministry of Strategy and Finance and KDI 2013).
This evaluation is unique in several ways. First, it was done in conjunction with the launch of the project in an attempt to provide real-time feedback to the project implementation agency. Health sector projects generally require years to complete, and evaluations done after completion provide useful lessons for future follow-on projects but not for current ones, potentially wasting both time and money.
Second, the evaluation research team was independent of the project implementation staff. The Impact Evaluation Lab of the KDI School of Public Policy and Management performed the analysis without being formally commissioned by the project financing or implementation agencies. Such a realtime impact evaluation by an independent entity has been advocated by several scholars and by the World Bank (World Bank 2014 and 2008, Thomas 2011, Lee 2011, Thomas and Tominaga 2010. However, it has not been done often. This study is, therefore, a rare pilot case in Korea.
Third, this study evaluates a health project that sought to improve the human capacity of UHS faculty members, rather than their equipment or physical infrastructure, as is often the case with development cooperation projects. Therefore, this project is similar to teacher education/training projects and technical assistance projects. With such projects, evaluations can be challenging since estimating project benefits in quantitative terms, using cost-benefit analyses or impact evaluation techniques, is difficult. Therefore, past evaluations of such projects have generally used qualitative or subjective analyses (e.g., statistical analysis of the trainers' or trainees' responses to such questions as whether the training program was useful or satisfactory). However, with such qualitative or subjective analyses, it is difficult to assure stakeholders that technical assistance programs are effective or efficient (SNUCM 2013, BMZ 2011, Marcano 2009, IMF 2005.
To overcome this deficiency, the current evaluation study adopts several innovative designs in pursuit of a more rigorous, evidence-based analysis. Given that the UHS project's ultimate goal is to improve the health of Laotians by enhancing the clinical knowledge and skills of hospital doctors, to be attained by advancing their academic achievement scores during their time as students at UHS, an evaluation should examine three hypotheses. The first of these determines whether the project improves UHS students' academic achievement scores. The second ascertains whether the improved academic scores of UHS students translate into better clinical performance of UHS graduates in hospitals. The third asks whether the enhanced clinical knowledge and skills in fact improve the health status of Laotians. This evaluation focuses only on the first hypothesis, as the project aims to advance UHS students' academic achievement scores by providing select UHS faculty members with a one-year training fellowship at the SNUCM.
This evaluation therefore adopts the unique strategy of evaluating the effectiveness of the teacher training project. It does not evaluate capacity improvement in teachers directly; instead it assesses the performance output of the trained teachers. Specifically, the improved academic achievement of the students taught by trained teachers is used as a proxy for the improved capacity to teach by the trained teachers. Many evaluation studies conducted thus far have tried to measure the trained teachers' improved capacity itself without much objective or quantitative success. Therefore, for this evaluation, a project implementation dataset containing data on the academic achievement progress of UHS students before and after the initiation of the project was compiled. Here, the UHS students are divided into two groups: one group taught by UHS faculty members trained at SNUCM under the project's fellowship (the treatment group), and the other taught by UHS faculty members who have not received such training (the control group). A comparison of the academic achievement levels of the two groups before and after the project fellowship will show whether the treatment group has in fact made greater academic progress.
The evaluation team used the difference-in-differences regression analysis method to test the hypothesis that the project fellowship training improved the academic achievement levels of UHS students. The difference-in-differences regression analysis method confirmed that the academic improvement achieved by the UHS students taught by project-trained UHS faculty members (the treatment group students) exceeded that of the control group, who were not taught by projecttrained faculty members. Various analyses support this finding, boosting the robustness of the test.
Although this project clearly helped to improve UHS students' academic achievement levels, there is no assurance that such an improvement will translate into better clinical performance by UHS graduates in hospitals. Thus, the evaluation team concludes that the ultimate objective of the project -improving the health status of Laotians -will likely not be attained solely by improving the human capacity of UHS faculty members and the academic achievement level of UHS students. Assurances with regard to achieving this objective will also require proof that the improved academic achievements of UHS students will translate into enhanced clinical performance by the same UHS students in hospitals upon their graduation. Such a test will have to be conducted as part of a conventional evaluation upon the completion of the implementation of the project.
The remainder of this study is organized as follows. The second section profiles the health statuses of Laotians and the status of UHS upon the inception of the project. The third section presents the results of the difference-in-differences regression analysis with the project implementation dataset, describing UHS students' academic achievement levels. The final section provides concluding remarks and recommendations.

A. The Health Status of Lao People
Lao PDR is classified by the U.N. as one of the least developed countries, with a 2013 per capita GNI of approximately $1,460. Although the total fertility rate now stands at 3.1 children and has been declining, Lao PDR has a young population with a median age of 21 years and with about 38% under the age of 15. Approximately two-thirds of the population lives in rural areas with a relatively sparse density (26 per square kilometer).
Total health expenditure per capita is estimated at $84. The health status of the population can be characterized as follows: a high mortality rate of 72 per 1,000 children under 5, a low rate of immunization against measles of approximately 50% of children aged 12-23 months, and an HIV prevalence rate of 0.1% of the population aged 15-49. Topping all causes of deaths, acute respiratory infections account for 20% of all deaths among children under 5 years of age. In addition, nearly 61% of children under age 5 suffer from diarrhea and receive oral rehydration therapy. The maternal mortality rate is also high at 220 per 100,000 live births.

B. The University of Health Sciences
The University of Health Sciences (UHS) is the sole institution of higher medical education in Lao PDR, and is a part of the Lao Ministry of Health. Before its reorganization in May of 2007, the faculty of UHS belonged to the National University of Laos under the Ministry of Education.
Currently, UHS has seven faculty groups, or divisions, including the Faculty of Basic Sciences and the Faculty of Medicine, the mainstay of medical education at UHS. It should be noted that the Dr. LEE Jong-Wook-Seoul Project is mainly partnering with these two faculty groups. Future collaborations between SNUCM and UHS may involve other faculty groups as well. In the academic year of 2009-2010, the total UHS student enrollment stood at 4,706, and the total faculty members stood at 215. The durations of the degree programs and the sizes of the student body and faculty vary across the constituent faculty groups. The details are presented in Table 1.
The number of incoming students has increased rapidly in recent years due to government policy. As a result, for the academic year of 2010-2011, there were 161 sixth-year students in the Faculty of Medicine, whereas there were nearly 400 firstyear students in the Faculty of Basic Science. The upsurge in student enrollment is mainly due to increases in the numbers of special students not selected through the competitive entrance examination. Except for the obligation to pay tuition and fees at a rate seven times higher, special students, who accounted for 55% of the total number of enrolled students in 2010, are treated equally to regular students. UHS adopted a new integrated curriculum in 2002 with support from the Canadian government. The new curriculum aims to train doctors so that they are capable of "working in hospitals or any other community facilities in Laos, with the adequate knowledge, skills and attitudes necessary to improve the health of people." Under the curriculum, fourth-year and fifth-year students spend their mornings at central hospitals for clinical training, while fifth-year students engage in in-field community practice for one month. In the sixth year, students receive day-long clinical training at hospitals. The quality of hospital services is poor, and it is difficult for students to learn good practices. Moreover, too many students are assigned to each professor to realize effective learning. On average, approximately 32 students are assigned per professor for clinical training, and occasionally the professors are unavailable to students.
The current educational environment at UHS can be described as minimal and in need of a major upgrade. A comparison between UHS and SNUCM in terms of basic aspects is striking. There are 55 professors in the combined faculty of the Basic Science and Medicine departments at UHS, whereas there are 503 professors at SNUCM. There are approximately 39 students per professor at UHS, much greater than the ratio of 1.3 at SNUCM. Classrooms are scarce at UHS, and the existing classrooms are inadequate for large classes because they have a flat floor. There are only a few laboratories with limited equipment and small class size capacities. The library has about 4,000 vintage books in different languages, and it can accommodate only 50 students in its reference room.

A. Methodology
Since schooling at UHS lasts at least six years, an excessive amount of time beyond the project implementation period would be required to test whether the UHS faculty members trained under the project indeed improved the clinical performance level of UHS graduates.
However, it takes relatively less time during the project to test whether the project-trained UHS faculty members improved their students' academic achievement levels. We therefore measured the project's impact on UHS students' academic achievement levels during the project implementation period.
For this test, we used a quasi-scientific trial design. The UHS students taught by the project-trained UHS faculty members were designated as a treatment (experimental) group. The students taught by UHS faculty members not trained under the project were designated as a control group. The academic achievement levels of these groups before and after the project were compared.
The difference-in-differences (DID) regression analysis method was used for the comparison. In addition, the fixed effects of the specific course and year observed were also controlled. The model used for the DID analysis method was as follows: ( The nomenclature is as follows: ijt Grade : The dependent variable represents the academic achievement of student i (1 to 5th-year students) for subject course j taken in year t (the official range is 0.0-4.0), TC : A dummy variable representing either the treatment or the control group (treatment group=1; control group=0), F_Year : A dummy variable representing either before or after the LEE Fellowship year (before= 0 for 2008-2009, 2009-2010, 2010-2011; after=1 for 2011-2012, 2012-2013, depending on the year the courses were offered), G : A dummy variable representing the gender of the students (male=1; female=0); S : A dummy variable representing the status of the students (regular status=1; special status=0) Age : Age of student i taking course j during t year, Experience : Teaching experience (number of years) of instructors teaching course j during year t, a : the constant term, b , c , d , f , g , h and k : Coefficient of the independent variables, and E : error term.
The key coefficient of this DID model is "d" of the interaction term ( TC*F_Year ). If it is positive, the trained faculty helped improve the students' grades.
The model was estimated with the pooled ordinary least squares method while controlling for the specific effects of each course and year, as the data are not panel data in a strict sense. Each student observed in different years does not represent the same student, and the years observed differ for each course.

B. Data and Sources
Altogether, 16 UHS faculty members received project training in 2010-2011 and 2011-2012. Of these, 10 faculty members taught courses upon their return. They taught 19 subject courses as part of a teaching team before (2008-2009, 2009-2010, and/or 2010-2011) and after (2011-2012 and/or 2012-2013) the training. During the period under review, all courses offered at UHS were taught by a team-teaching method. The remaining six faculty members, who did not teach either before or after the Dr. LEE Fellowship award, were excluded from the definition of the treatment group.
The UHS faculty members who did not receive training taught approximately 23 courses as part of a teaching team before and after the project period (excluding those courses offered for sixth-year students and languages and social science courses). These types of courses tended to be taught by more experienced faculty members who were not considered for the Dr. LEE Fellowship, with a focus on improving the capacity of relatively young faculty members first. In fact, only one course was eligible for the control group, as the data representing students' characteristics were not available for the remaining courses.
All data were sourced from the UHS administration during 2010-2013. The descriptive statistics are as follows:

C. Estimation Results
The results of the estimation are summarized in the following table. Equation (1) was estimated first without controlling for the fixed effects of each course and year (1B).
In the estimation of the model without course and year fixed effects controlled (1B), the coefficients of all variables were significant, except for the age variable, which was deleted by the computer program during the estimation process due to possible multi-collinearity or missing observations. During the entire period observed, male students achieved less than female students, regular students performed better than special students, the grades of the treatment group declined, students taught by more experienced instructors performed worse, and the grades of all groups after the fellowship period declined at the one percent significance level. However, when those factors were controlled, the grades of the treatment group were higher after the fellowship award relative to those of the control group by 0.29 percentage points at the one percent significance level. Therefore, we can attribute the higher grades of the treatment group to the Dr. LEE fellowship.
In the estimation of Equation (1) with course and year fixed effects controlled (1A), coefficients did not change significantly. The coefficient "d" of the interactive term changed from 0.29 to 0.33 with the same degree of significance, indicating that the treatment group (students taught by former Dr. LEE Fellows) had higher scores after the initiation of the fellowship program in 2010. All other coefficients maintained the same sign and degree of significance, except that the gender variable coefficient become more significant at the one percent level, the F_Year variable became statistically insignificant, meaning that the scores of all students (the treatment and control groups combined) did not change before and after 2010, and the TC variable was deleted due to the time-invariable nature of the variable. Therefore, the robustness of our estimation of Equation (1) was enhanced when it was tested with and without controlling for course and year fixed effects.

D. Further Robustness Tests of the Estimation Results
It is important to note several caveats pertaining to this study thus far. First, up to this point, only one course is considered as a control group due to the difficulty in obtaining data on each student's characteristics during the period before the Dr. LEE Fellowship was offered. We can increase the number of subject courses taught by the control group (non fellows). Instead of observing individual students' academic achievement levels, therefore, the average score in each of all subject courses offered by UHS can be observed before and after the Dr. LEE Fellowship award (except for the language and social studies courses). We can also control for the fixed effects of the course and year observed.
Second, we can refine the definition of the treatment group. The basic estimation model would define the treatment group as the courses taught by the former Dr. LEE Fellowship awardees as part of the teaching team (TC). Another way to define the treatment group is to use the share of the former Dr. LEE Fellowship awardees out of the total number of teachers on the teaching team for each course (SFT). Still another way is to define the treatment group as the share of former Dr. LEE Fellowship awardees' teaching hours out of the total number of team teaching hours for each course (SFH). We can then observe each course once before 2010-2011 (either 2008 or 2009) and once after 2010-2011 (either 2012 or 2013) when the LEE Fellowship was awarded, depending on the year each course was offered. The revised models, therefore, can be specified as follows.
(2)   jt j t j t j t j t j t Average Score =a+bTC +c F_Year +d TC *F_Year +E Average Score =a+bSFT +c F_Year +d SFT *F_Year +E Average Score =a+bSFT +c F_Year +d SFH * F_Year +E All courses offered by the UHS were included (a total of 46 courses), excluding social studies and language courses. They were divided into 24 treatment group courses taught by the Dr. LEE Fellowship beneficiaries and 22 control group courses taught by the non-fellows. However, the treatment group courses (TCs) are expressed here by "1" if the course was taught by Dr. LEE Fellowship recipients (Equation (2)) as well as by the share of the Dr. LEE Fellowship recipients out of the total number of team teaching members for each treatment course (SFT) (Equation (3)), or the share of hours taught by the Dr. LEE Fellowship awardees out of all team teaching hours for each course (SFH) (Equation (4)). The average grades (scores) of the treatment and control group courses were observed one year in each case before the Fellowship award (either 2008-2009, 2009-2010, or 2010-2011) and once after the Fellowship award (either 2011-2012 or 2011-2013), depending on the courses offered. The basic DID model to control for the fixed effects of each course and year was used to confirm whether the grades (scores) of the treatment group courses increased more than those of the control group courses since the Dr. LEE Fellowship award. All of the other variables of the models are defined in the manner of Equation (1). A summary of the statistics is given below.    The estimation results of the DID analysis with TC, SFT, and SFH variables are presented in table 7. As shown in model (1) or (2), the average score of the treatment group courses (TC) were lower than that of the control group courses. However, after the LEE Fellowship training opportunity was given during 2010-2011, the grades of the treatment group (courses taught by Dr. LEE fellows trained at SNUCM) were higher than those of the control group courses (taught by nonfellows) by 0.205 or 0.254 percentage points at the one or five percent significance levels ( TC*F_Year ). Therefore, we can attribute the treatment group's higher grades after the LEE Fellowship award to the training program at SNUCM.
Moreover, when the treatment group was defined in a more refined way (models 3-6), the coefficient of the interactive terms ( F_Year*SFT and F_Year*SFH ) becomes greater. This means that since the LEE Fellowship award, the average grades of the treatment group courses have become higher (by 1.068 or 0.671) than those of the control group courses. Therefore, the robustness of the estimation results based on each student's achievement score (equation (1)) has been confirmed positively. (Only when the SFH variable was used, the coefficient of the interactive term was not significant statistically even at the 10% level.).

IV. Conclusions and Recommendations
In early 2011, a KDI School research team launched a real-time impact evaluation of the University of Health Sciences (UHS)-Dr. LEE Jong Wook-Seoul Project. The design of the project is based on the premise that the project's final outcome, i.e., improvements in the health status of Laotians, will be achieved by enhancing the clinical performance capacity of the practicing doctors, to be attained by increasing academic achievement scores at UHS. Therefore, the main focus of the real-time impact evaluation was to assess whether the project's one-year fellowship training of select UHS faculty members at SNUCM has indeed resulted in advancing the academic achievement scores of the UHS students during the project implementation.
Our real-time impact evaluation team conducted difference-in-differences regression analyses and showed that the project improved UHS students' academic achievement levels. In the analysis of the academic achievement scores of students in their first through their fifth years obtained before and after the Dr. LEE fellowship training periods, the UHS students did achieve a greater advancement in the courses taught by the project fellowship recipients as compared to other subjects taught by the non-fellows, even after controlling for the students' gender, age, and status, and for the fixed effects of years or courses.
It remains to be determined several years after the project has been implemented whether these UHS students taught by the LEE fellowship recipients would perform better clinically in public hospitals compared to UHS students taught by non-fellows in the future. The follow up project (the Second UHS-LEE Jong-Wook-Seoul project) should finance the collection of data on the academic achievement and clinical performance levels of UHS graduates working at public hospitals who were taught by SNUCM-trained faculty members so that a rigorous ex-post impact evaluation can be carried out in due course.