A Practical Framework for Evaluating Health Services Management Educational Program : The Application of The Mixed-Method Sequential Explanatory Design

© 2015 The Authors. This is an Open Access article distributed under the terms of the Creative Commons Attribution (CC BY), which permits unrestricted use, distribution, and reproduction in any medium, as long as the original authors and source are cited. No permission is required from the authors or the publishers. Article History: Received: 8 Feb 2015 Accepted: 1 June 2015 ePublished: 4 July 2015


Introduction
Medical or health services managers, also called healthcare administrators or healthcare executives, are responsible for providing and commissioning the local healthcare through planning, directing, and coordinating services. 1,2s healthcare managers, they are required to manage the costs, delivery and quality of healthcare services once liaising and cooperating with clinical and non-clinical staff. 1,2In line with the revolution in healthcare, health services managers must be able to keep up with the changes in laws, regulations, and technology in order to improve the efficiency and quality in delivering healthcare services.With that in mind, health services managers are key players in hospitals and community health services where they take several roles, including human resources, staff management, clinical management, facilities management, and finance. 1,2ver the past decades, health services management (HSM) programs have been widely established to provide hospitals and health-related systems with skilled, professional managers to address health and community needs and expectations.It becomes essential that these training programs are not only retained, but also updated according to the advances and changes of health and community needs. 3It is also important to ascertain the quality of HSM educational programs in order to improve or renew them. 3One way to do this is to determine the value and effectiveness of the HSM programs through systematic and planned activities based on the concepts of program evaluation. 4 Program evaluation is defined as a systematic operation to study the merit, quality, and worth of some or the whole components of the program. 3,5Depending on the program's definition, evaluation might include a program's design, learning environments, teaching-learning strategies, resources and materials used in instructional procedures. 3rogram evaluation is an essential part of improvement, renewal, and long-term achievement. 4It provides continued development and reveals what is effective and what is not.So, evaluation which results in the improvement of an HSM program is highly crucial if quality and effectiveness are to be achieved and maintained. 4HSM program evaluation helps identify the necessary improvements and modifications to be made in the content, teachinglearning procedures, learning facilities, staff selection , and the development of program goals and objectives. 3It also provides evidence of the program's effectiveness for administrators and policy makers. 6 comprehensive evaluation of the HSM program in terms of its effectiveness should be based on different sources of evidence and different measures. 4,5To do this, qualitative research methods as well as statistical analysis and quantitative measures should be used to provide in-depth analysis and information.Depending on the methods applied, and the program evaluation approaches, a huge range of evaluation models are identified. 6The CIPP (Context, Input, Process and Product) evaluation model is one of the most widely-implemented models. 6It is developed to systematically guide both evaluators and administrators in asking questions of vital importance in an evaluation process and in doing an evaluation at the beginning of a program (context and input evaluation), while the program is in progress (input and process evaluation) and at the end (product evaluation). 68][9][10][11][12][13][14][15] Originally, evaluating the quality and effectiveness of the HSM program requires a systematic evaluation of all aspects related to current processes and activities.However, the current evaluation approaches and their uses on HSM programs seem to lack a comprehensive and explanatory approach.Therefore, the purpose of this study is to synthesise and develop a framework to evaluate the HSM program.Added to this synthesis are the explanations and experiences of HSM academics and alumni.Our aim was to indicate how practical our framework could be applied in the evaluation of the HSM program.The framework provided by this article is intended to help those administrators, policy makers and teachers who are involved in planning, directing or evaluating the HSM program.

Study Design
The study was intended to evaluate the Health Services Management program using a CIPP evaluation framework.This research, conducted during April-July 2013, followed a mixed-method sequential explanatory approach in which data were collected through a CIPP survey and semi-structured interviews.Mixed methods are defined as the tool of collecting, analyzing, and integrating or mixing quantitative and qualitative data during the research process within a single study to reach a better understanding of the research problems. 16The mixed method sequential explanatory design is one of the most popular research designs in which first quantitative data and then qualitative data are collected and analyzed.This study consisted of two main phases of research design.In the first phase, the HSM program was evaluated using self-administered questionnaires drawn from CIPP evaluation framework (Table 1).In the second phase, indepth research was conducted to explain and analyze the first quantitative findings through semi-structured interviews.

Participants
In this study, samples were selected through different methods.In phase 1, participants included 10 faculty members, 64 students and 90 alumni.The majority of respondents were female (n=124).All faculty members and students taking part in the program were included in the study.Alumni were those who had graduated during the last five years.In phase 2, a purposeful sampling method was used, and participants included 8 faculty members, 15 alumni and 5 mentors.Of all the participants in phase 2, 13 were female (46%).

Instrument
The CIPP survey A pilot study was conducted to assess the validity of the designed questionnaires (face, content and construct validity).In this case, the questionnaires were phrased appropriately for easy answering and lack of ambiguity.The Cronbach's alpha coefficient was applied for reliability purposes and it was reported α=0.85, α=0.75 for the students and alumni, respectively.The conceptual framework that guided the development of the questionnaires was based  1).The questionnaires consisted of two parts: the first part took into account the socio-demographic characteristics of respondents and the second part was designed in view of the CIPP evaluation format (Table 1).The questionnaires were designed on a 5-point Likert-type scale, ranging from completely disagree (scoring 1) to completely agree (scoring 5).

Semi-structured Interviews
In phase 2, in-depth semi-structured interviews were used to analyze and explain the quantitative findings.In addition to the quantitative findings, the following sources were used for collecting the data in order to understand the program's issues better: (1) in-depth semi-structured interviews with participants; (2) participants' responses to survey questions in the first, quantitative, phase.

Data analysis
In phase 1, descriptive statistics (mean and standard deviation) were used to analyze the survey data.The frequency counts helped analyze the participants' answers to separate items on each of the five survey scales.SPSS 16.0 was used for these tests.
In phase 2, interviews were audio-taped and transcribed verbatim.Duration of interviews ranged from 30 to 45 minutes based on the conditions and participants' desire.
The thematic analysis of the text data was done by using NVivo N8, which is a qualitative software for data storage, coding, and theme development.Credibility of the findings was secured by member checking and academic advisor's auditing.

Demographics
The HSM program was evaluated through CIPP surveys sent to faculty members, students, and alumni.In total, 10 faculty members (4 females, 6 males), 64 students (49 females, 15 males) and 90 alumni (71 females, 19 males) participated in the study and returned the questionnaires (response rate: 75%).The average age was reported as 35, 21 and 25 for faculty members, students, and alumni respectively.

Phase 1: The CIPP Survey
The survey questions focused on 7 main areas, including program goals and objectives, curriculum content, educational facilities, process and activities, program performance, graduate skills and overall impression of the program.With regard to program goals and objectives, 8 respondents (8%) pointed out that the objectives were met at the end of the program.However, the majority of respondents (74%) stated that the program goals and objectives were not clearly defined.Generally, 57.6% of respondents were dissatisfied with the program goals and objectives.Table 2 illustrates the satisfaction of respondents regarding HSM program goals and objectives.
In terms of the curriculum content, most students and academics (59.5%) reported that the curriculum contained outdated information.However, 33% of respondents were undecided about whether the curriculum content followed the program goals or not.In terms of educational facilities, most respondents (n=48) reported that the classes were well equipped and learning resources were well prepared (n=41 The major limitation of the HSM curriculum was relevant to the imbalance between the content and initial program goals.Curriculum content is outdated and it does not place an equal emphasis on theory and practice (Table 3).It seems that teaching-learning activities are well organized and students are well involved in different activities.
However, new teaching methods were recommended to improve the current status (Table 4).The curriculum content seems to be the main source of dissatisfaction among alumni and faculty members.The curriculum content does not emphasize students' skills and attitude.Further explanations are demonstrated in Table 5.

Discussion
The purpose of this study was to synthesize and develop a framework to evaluate the HSM program, following a mixed-method sequential explanatory approach in which data were collected through the CIPP evaluation framework and semi-structured interviews.The HSM program was evaluated in two phases.In phase 1, the CIPP evaluation framework was prepared in line with the relevant literature review and then it was sent to students, faculty members, and alumni in order to evaluate the HSM program.The analysis of the responses by 10 academics, 64 students, and 90 alumni revealed several issues for stakeholders and decision makers to be taken into account once renewing and revising the HSM program, including the program goals (context), curriculum content (input) and graduate skills (product) which are highly relevant to the program's quality. 14ontext evaluation is an important part of the evaluation process in which background, knowledge and beliefs about the program, the program's objectives and its role in society are explored. 17In this regard, the main purpose of the HSM program is to provide hospitals and health-related systems with skilled, professional managers to address health and community needs and expectations. 1 -The curriculum is content-oriented -The curriculum should be turned on its head, with students starting to think like a manager from the day they enter school -The curriculum should be revised and updated regularly -The curriculum should be integrated between theoretical and practical lessons -Students should be introduced to applied lessons alongside the theoretical lessons from the beginning of the program -Faculty members should be involved in curriculum development -The need-based approach is an alternative to its implementation

What content should be included in the curriculum?
-There should be iterative revisiting of topics throughout the course -Topics relevant to Strategic planning, policy and decision making in health, operational research in health settings, accreditation and clinical governance, and project management are recommended to be included in the curriculum  The curriculum is outdated and do not provide fresh content to students -The curriculum does not address the attitude of being a manager to students - The imbalance exists between the theoretical and internship contents - The internship experiences are not supervised and evaluated properly -Lack of experienced and knowledgeable mentors during internship experiences - The hospital departments are not aware of the program goals - The program actually does not make a difference in students´ knowledge and skills and most of the learnt lessons will be forgotten over time - The program neither satisfies the students´ needs nor messages the health setting expectations and needs - The employers view the performance of former students and believe that they don't have the expected competencies - The program stakeholders are not aware of the substantive expectations and needs of health settings the current program should be revised rigorously and updated by health setting challenges and needs -By removing older topics from the curriculum, the course becomes progressively able to produce competent managers - The course contributes to the disillusionment and demoralization of students by deadening their initial enthusiasm and fails to prepare them adequately as professionals Table 5. Product evaluation of HSM program and in depth-analysis public management, privatization and decentralization of government services, attention to accountability and transparency have affected HSM approaches in western countries. 17However, European countries have rather a different approach and pay more attention to policy.Therefore, it becomes apparent that programs must be compatible with social, cultural and even geographical contexts to address local needs.It seems that the HSM program in Iran has not been well designed based on the local healthcare challenges and needs and this has led to the current mismatch that exists between the program objectives and local needs and expectations.Thus, are gular needs assessment is highly recommended to determine the community needs, assets, and problems and to answer the question, "what needs to be done?". 17n our study, curriculum content was considered and evaluated as the key input of the HSM program.In a system-based approach to education, inputs are defined as the key resources needed to run the program. 5Input evaluation is intended to determine what resources are available and what alternative strategies and plans should be considered to meet the program needs. 5It seemed that the HSM curriculum content was the other key limitation of the HSM program.Most respondents (59.5%) reported that the curriculum contained outdated information.However, 33% of respondents were undecided whether the curriculum content followed the program goals or not.
It was also assumed that the curriculum content did not address community needs.This problem was also observed in medical education in Iran, in which the clinical modules were not based on community needs. 18It was also claimed that there was no integration between theoretical and practical lessons in the HSM program.As a general concern, the HSM program is designed according to medical education in Iran.So, these problems are rather common concerns in most health and medical sciences in the country.It is worth referring to the super system in which health and medical sciences function.
Each training program is embedded in a super system 5 that holds and manages all health and medical sciences together.
The Ministry of Health as a super system has established and monitored the national medical education system in Iran.It is obvious that the challenges and limitations of this super system could influence the quality and performance of the embedded programs.Evidence shows that less attention has been paid to curriculum development in Iran 18 , and that is why the HSM program as an embedded program in national medical education was evaluated as an outdated program.According to our findings, regular content revision and an emphasis on both theory and practice were recommended to improve the program.Curriculum content has been identified as the main source of dissatisfaction in several studies. 8,14,18 6 In product evaluation decisions are made on a regular basis.Questions such as, "What results were obtained?"and "What should be done with the program?" are important in judging the worth and attainment of the program. 5The impact of product evaluations considers the longer-term effects of the program and the extent to which the program eliminated or reduced the students' needs. 4Product evaluation in the HSM program revealed several issues.First, graduates' skills were far from the program goals and students did not have the expected skills and competencies at the end of the program.Second, the program actually did not make a difference in students' knowledge and skills and most of the learned lessons were forgotten over time.
In addition, the employers viewed the performance of former students and believed that they did not have the expected competencies to set out on a career path into the healthcare field.It seems that the HSM program has several limitations in its design and development.
Our findings cast some doubt on the quality of the HSM program.It appears that there is a need to train managers to respond to the needs of stakeholders and healthcare providers.The process that the program uses should insure that all graduates fulfill the requirements of their future professional competencies and that they are accountable to the healthcare system.Therefore, the HSM program should be revised rigorously and a community needs assessment, content renewal, and curriculum development are recommended to improve the program.In our study, the HSM program was evaluated according to CIPP (Context, Input, Process, and Product) evaluation framework.In this regard, students, instructors, and alumni were involved to explore their perspectives about the program.Further research is recommended to explore the program's stakeholders and healthcare providers' expectations and also their comments on the HSM graduates' performance and skills.Since evaluation strategies or designs do not meet the needs of all programs or all decision makers and stakeholders, further evaluations are recommended.These evaluations should be adapted to the program's history and stage and should be performed in a manner "that is appropriate for program stewardship and decision makers".

Table 1 .
CIPP evaluation framework applied to the HSM program on 7 factors and 30 indicators (Table the program goals and objectives was the imbalance between the community needs and the program goals.It seemed that the program goals were far away from the reality of community needs, and most were time consuming and did not reflect the knowledge, skills, and practice of students.It was suggested that the program planners conduct regular needs assessments and focus groups to update the program goals and match them with community needs and expectations.Further explanations and suggestions are illustrated in Table2. ). Regarding the process and activities, most respondents (n=41) reported that students were actively involved in classroom activities.The majority of respondents (n=43) pointed out that the instructors implemented appropriate teaching strategies.Almost all respondents (78.3%) agreed that the instructors were cooperative and interested in involving students in research activities.However, 58.1% of respondents stated that the instructors did not use problem-based subjects in class.Phase 2: In-depth semi-structured Interviews To identify the potential causes and obtain suggestions from academics, mentors, and alumni regarding improvements to the HSM training program, in-depth semi-structured interviews were conducted.The main barrier identified for achieving The study results yielded several important lessons and suggestions for the future of the HSM program.The framework provided by this article is intended to help those administrators, policy makers and teachers who are involved in planning, directing or -The objectives follow a hospital-based approach Any suggestions to improve the current status?-A periodic assessment of community needs and expectations towards the program goals/objectives -Regular focus groups should be scheduled with program stakeholders to discuss the barriers and suggest solutions -The objectives of internship experiences should be based on the fundamental requirement of the community needs

Table 2 .
Context evaluation of HSM program and in-depth analysis evaluating the HSM program.

Table 3 .
Input evaluation of HSM program and in-depth analysis -The current teaching-learning activities are satisfying but shifting to new teaching strategies (working in groups, problem based learning, learning in small groups) may result in the improvement of the current satisfaction and quality of the program -Teachers are enthusiastic towards research responsibilities to make students competent in research

Table 4 .
Process evaluation of HSM program and in-depth analysisHow does the program work and what are its outcomes?-Theacquired skills of graduates are far from the program goals and students do not gain the expected skills and competencies at the end of the program - Therefore, more attention should be paid to curriculum development and content if the quality and effectiveness of programs should be maintained and improved.It seems that the issues highlighted in this study are relevant to the lack of an innovative curriculum, considering the present healthcare challenges, expertise and professional competencies.It is also expected that new modifications on national medical education systems in Iran will result in the improvement and development of the current program.In this study, graduates' knowledge and skills were evaluated as the program's products.Product evaluation is intended to identify the program achievements and ascertain to what extent the students' needs were met.
17,19ConclusionBased on the HSM program, training skilled and professional managers for hospitals and community health services is a genuine and valuable effort.It is important to ascertain the quality of the program if the effectiveness and quality need to be maintained and improved.Therefore, the purpose of this study was to synthesize and develop a framework to evaluate the HSM program.This evaluation study reveals to what extent the HSM program objectives are being addressed.Learning barriers, further suggestions, and modifications are provided to improve the quality of the program.We believe that our study will serve as an initiative for future research on the systematic evaluation of HSM program.The CIPP evaluation framework helped researchers focus on the context, input, process and product of the HSM program, taking into account the perspectives of different students, alumni, faculty members, and mentors.CIPP evaluation is dedicated to provide researchers with some guidance on how to design, conceptualize, implement, and validate their research to ultimately produce more meaningful study outcomes.The framework provided by this article is intended to help those administrators, policy makers, and teachers who are involved in planning, directing or evaluating the HSM training program.