A Critical Appraisal Model of Program Evaluation in Adult Continuing Education

This paper describes a critical appraisal model of program evaluation that was developed specifically for a university continuing education context (i.e., credit and non-credit programs designed to meet the personal and professional development needs of adult learners). The articulation of this model is a result of the first Prairie Symposium on Research on University Continuing Education, held in June 1999, and of ensuing discussions about the need for a proactive, research-based, and adult learner-oriented approach to evaluating programs. The paper begins with a brief overview of the conceptual framework of the model, followed by a discussion of the process and content issues deemed important within a university continuing education context. The Prairie


INTRODUCTION
One of the actions recommended at the Prairie Symposium on Research on University Continuing Education, held in Winnipeg on June 3-4, 1999, was to develop a model of program evaluation or critical appraisal grounded in a developmental and collaborative approach. Symposium participants believed there was a need for an approach that is accountable and responsive to adult continuing education stakeholders (i.e., learners, their sponsors, and relevant professional accreditation bodies). An additional factor was the desire to clearly articulate a process that distinguishes adult continuing education evaluation and program development procedures from those used by traditional faculties. It was decided that participating institutions would subsequently annually identify one program among them to be reviewed, using this model, by a team made up of their continuing education staff. The broader term "critical appraisal" was chosen to connote an evaluation process that moved from a "what is" to a "what can be" perspective in a future-oriented and critically reflective manner. The larger objective of this targeted and empirically based research activity, which would ideally include the publication of each program model is then illustrated in detail through a case study of the approach used by one university faculty of continuing education to evaluate its graduate program in workplace learning. The paper closes with a discussion of the applicability of this model to other adult continuing education programs.
est suivi d'une discussion du processus et des questions de contenu jugées importantes à l'intérieur d'un contexte d'éducation permanente universitaire. Ensuite, en utilisant une étude de cas de l'approche qu'a empruntée une faculté d'éducation permanente universitaire pour évaluer son programme d'études graduées en apprentissage en milieu de travail, le modèle est détaillé. L'article termine avec une discussion de l'applicabilité de ce modèle à d'autres programmes d'éducation permanente pour adultes. review, would be to improve the practice of program development in adult continuing education in Canada.
The proposed model was to be grounded in an action-research approach, with a focus on the creation of an innovative solution to a real world problem. It was also to incorporate the active participation and collaboration of members of a research (or critical appraisal) team, translate the research findings into practice of a future-oriented nature, and recognize the situational nature of the appraisal process (Sloane-Seale, 1999). The model, as envisioned, would provide a template that would encompass both the process (i.e., how the evaluation of the program would take place) and the content (i.e., what aspects of the program would be evaluated) of the critical appraisal process.

CONCEPTUAL FRAMEWORK OF THE CRITICAL APPRAISAL MODEL
The approach to the evaluation phase of program planning is fundamentally related to each individual's philosophy of program planning (Brookfield, 1986;Worthen, Sanders, & Fitzpatrick, 1997). In his comparison of the predominant world views of modern social science research in Canada (positivism, interpretive humanism, and radical structuralism), McLean (1999) outlines several philosophical, methodological, and ethical issues to consider in program design, and the implications of each for research within/on university continuing education programming. In this paper, the proposed critical appraisal template most closely aligns with the interpretive humanist and positivist world views in terms of its ontological, epistemological, and ethical assumptions and its methodological assertions (see McLean, 1999). However, the flexible and process-oriented nature of the template will allow any institution using it to be guided by the particular meta-theoretical assumptions decided upon by its critical appraisal team.
In addition to different world views of research/program planning, there also exist alternative approaches to evaluation that imply alternative methodologies (see Worthen, Sanders, & Fitzpatrick, 1997). Symposium participants expressed the desire for both an empirically based and learnercentred approach to program evaluation. Given this, the proposed critical appraisal template contains aspects of both the objectives-oriented and participant-oriented approaches in terms of the purposes of, and the criteria for judging, evaluations.
To locate the critical appraisal template within the field of adult learning, it is useful to recall Brookfield's (1986) recommendation that three particular models of evaluation appear most likely to qualify as candidates for a uniquely adult education evaluation framework: participatory evaluation, perspective discrepancy assessment, and andragogy. Participatory evaluation (which allows adult learners to assume control for the evaluation of their learning) and perspective discrepancy assessment (which assumes that the educational process is best understood by examining how participants perceive it) both appear to be consistent with the interpretive humanist perspective. Andragogy (which aims to move participants towards taking responsibility for the educational process and their own learning) appears consistent with the radical structuralist perspective. Although Brookfield does not endorse what he described as the "Tylerian" aspects of the positivist perspective, he does consistently incorporate aspects of an objectives-and results-oriented approach in his own program planning "best" examples, thereby implicitly endorsing such an empirically based approach. Just as the three models described by Brookfield are learner-centred, so too is the critical appraisal model described here.
Program evaluation is one phase of the broader program planning process. Of the numerous models currently being used by adult educators, this paper grounds the proposed critical appraisal model within Caffarella's (1994) interactive model of program planning. Caffarella developed this model from the literature on systematic processes for evaluating programs for adult learners and it is both systematic and non-linear in nature. The critical appraisal model described in this paper closely resembles the evaluative component of Caffarella's model, with the following additions.
1. The purpose of the evaluation and how the results are used are to be grounded in a utilization-focused approach that uses the entire evaluation process as a framework for improving the program under review (Patton, 1997).
2. The conceptual framework underlying the evaluation process is to be clearly grounded in both adult education and organization development evaluation philosophy and practices (Selman, 1999;Vella, Berardinelli, & Burrow,1999).
3. Teachers' learnings, students' and colleagues' perceptions (Brookfield, 1995), discussions with peers, consultations with others, and feedback from learners (Cranton, 1996), as well as key practices derived from organizational program evaluation (see Wholey, Hatry, & Newcomer, 1994;Worthen, Sanders, & Fitzpatrick, 1997), are to form the guidelines for institutional critical self-reflection. Institutional critical self-analysis is to be defined as the extent to which intended users of the evaluation findings are actively involved in conceptualizing, applying, analyzing, and synthesizing the findings in order to create a guide to further action. It entails the examination of the original purpose and assumptions upon which the program was developed, and critical reflection on whether or not the program's best interests are served by continuing with this original underlying framework.
4. Strategies for engaging users and program stakeholders in an ongoing process of program/organizational development to keep them actively involved in program decision-making are to be added (Patton, 1997).

PROCESS AND CONTENT ISSUES IN PROGRAM EVALUATION
Process Issues Patton's (1997) utilization-focused approach begins with the premise that evaluations should be judged by their utility and actual use. It is primarily concerned with how results can be used by the people affected by the evaluation to address real issues within the program being evaluated. Patton's approach does not advocate any particular evaluation content, model, method, or theory, but rather describes a process for helping intended users select the most appropriate content, model, method, theory, and uses for their particular situation. Thus, utilization-focused evaluation can include any evaluative purpose (formative, summative, developmental), any kind of data (quantitative, qualitative, mixed), any kind of design (naturalistic, experimental), and any kind of focus (processes, outcomes, impacts, costs, and cost/benefit). It is a process for making decisions about these issues in collaboration with an identified group of primary users, focusing on their intended uses of evaluation (Patton,pp. 20 & 22).
Utilization-focused evaluation is grounded in a number of fundamental premises, including: • commitment to intended use by intended users should be the driving force; • careful and thoughtful stakeholder analysis involving political sensitivity and ethical judgements should be done; • evaluations must be designed and adapted situationally to take into account such things as community variables, organizational characteristics, the nature of the evaluation, evaluator credibility, political considerations, and resource constraints; • the intended users' commitment to use should be nurtured and enhanced by actively involving them in making significant decisions about the evaluation.
Within a university continuing education context, this approach would actively involve all of the stakeholder groups affected by the program undergoing evaluation at every key decision point along the way. The evaluators have both process and content roles to play in this exercise, as they facilitate the evaluation process, produce a high-quality product, and teach the evaluation users/stakeholders how to use the evaluation findings.

Stages of a Utilization-focused Evaluation Process
The "evaluator" in Patton's (1997) model refers to the person(s) identified as responsible for conducting the evaluation, often an externally contracted expert in the business world. Within a university continuing education context, this person(s) will typically be the chair of the program evaluation committee. "Intended users" in Patton's model refer to the key stakeholders of the program being evaluated, such as sponsoring agencies, the agency delivering the program, program participants, and community members benefiting from the program. Within the university context, intended users will include faculty and staff involved in the program's design, delivery, and administration, as well as the students enrolled in it and their employers or clients. Patton's model includes 12 steps that are contained within five evaluation stages, beginning with the identification of key stakeholders who will then make up the membership of the evaluation committee. Committee members then collaboratively determine the focus and intended use of the evaluation; decide on the method, measurement, and design of the project; interpret the findings and make recommendations; and, finally, determine the dissemination of the evaluation report.
The evaluation process may appear linear. However, within a situationally responsive context, it is necessarily circular and iterative as the process unfolds and new stakeholders or new questions emerge. The point of greatest vulnerability for this approach lies in the turnover of primary intended users and the need for their active engagement at every point along the way. Generally, this is less of a concern within an academic as opposed to a business context, because faculty, staff, and students tend to remain relatively stable or accessible over a typical program-evaluation time period. Vella, Berardinelli, and Burrow (1998) set out three criteria for effective and useful evaluations. That is, they must be carefully done by those best qualified to judge effectiveness, provide more returns than costs, and be accomplished in a manner that fits all concerned-the organization, program, educators, and learners. They described evaluation as a process of accountability that connects evaluation to program planning and engages learners as partners throughout. Their criteria are grounded in adult education and learning principles and in program evaluation philosophies and procedures practised in educational and non-educational organizational contexts. They believe that effective evaluation must be objective (while focusing on both outcomes and process) if clear evidence of whether, in fact, a program is leading to desired change is to be obtained. It should also identify the important elements of an educational program in order to allow for future planning, match the organizational philosophy so that results will be meaningful, and use measures that will not impose significant burdens on either the learners or the organization. They refer to the framework they use as the accountability process because, although it focuses on evaluation, it clearly demonstrates the linkage between program evaluation and program design.

Content Issues
The theoretical framework of the accountability process is diagrammed below. It illustrates how each element (beginning with the identification of the purpose of the program being evaluated) flows from the previous one as a result of systematic planning that considers the results expected (in terms of measurable organizational improvement) from an education or training program.
Program planning identifies and anticipates the relationships of these elements: Program evaluation measures the effectiveness and relationships of these elements.
Each of the three outcomes (learning, transfer, and impact) is an independent measure. Learning is defined as changes in learners' knowledge, skill, and attitudes that result from the program. Transfer is the learning that is applied in the learners' work after completion of the education or training program. Impact is the improvement in the performance of the learners' organizations as a result of the learners' work. Vella, Berardinelli, and Burrow (1998) believe that adult education practitioners, working with other stakeholders, are responsible for determining what performance measures will be used for each of these results. The critical appraisal model described in this paper considers all three of these results to be important considerations in the design of the evaluation data-collection tools and processes.

Political and Contextual Issues
Evaluations are inherently political due to the differing values, perceptions, and politics of those involved, the requirement that data be collected, classified, categorized, and interpreted, and the fact that decisions/actions follow from evaluation findings. of abstract planning models, while ignoring the political realities of actual planning practices. They suggest that the conventional literature marginalizes the organizational and social contexts of planning practice, and state their objective as the development of a theory that accounts for what really matters about the actions of adult educators in the everyday world (1994a). Program planning practice is defined by them as a social activity in which people negotiate personal and organizational interests to construct adult education programs (1994a). To understand program planning, Cervero and Wilson (1994a) conceptualize power, interests, negotiation, and responsibility, and they argue that adult educators should have an ideal interest in nuturing a substantively democratic planning process. Cervero and Wilson (1994a) also argue that responsible program planning first entails anticipating how existing relations of power are likely to support or constrain a substantively democratic planning process and then acting in ways that nurture such a process. They believe there are five groups of people whose interests always matter in planning programs: learners, teachers, planners, institutional leadership, and the affected public.
From the evaluation process perspective, Patton (1997) offers some guidelines for forming and chairing the evaluation committee. These guidelines stress the importance of carefully considering its membership in terms of stakeholder representativeness, its ability to influence, and its commitment to the task. Another important consideration is the groupprocess skills required by the committee chair. This person must not only be very sensitive to members' individual political issues but also be able to use participatory decision-making at key steps of the evaluation process and maintain an empirical/objective focus.
The case study that follows describes the critical appraisal process used by one faculty of continuing education. It closely followed the basic premises, principles, and processes of Patton's utilization-focused process model, while integrating process, content, and political issues specific to a university continuing education environment (see Appendix 1). A primarily distance-delivered graduate program in workplace learning was chosen as the case study because the author of this paper also chaired the evaluation committee for that program (which became the trigger for the development of the critical appraisal model). It is important to note, however, that the principles and process described here are equally applicable to any program committed to being accountable to its key stakeholders and to utilizing the evaluation findings to improve the program under review. Due to the limited availability of some of the stakeholders in the case study, the involvement of all stakeholder groups in each of the critical appraisal decision-making steps was neither as complete nor as continuous as desired. However, this case study is only one illustration of how the critical appraisal model can be used to evaluate an adult education program offered within a university continuing education context. The specific methods, measures, and decisions made will vary, depending on each program's unique characteristics and the institutional circumstances.

The MCE Program
In the mid-1980s the Faculty of Continuing Education at the University of Calgary began to research the growing demand for a graduate program in continuing education to address the requirements of workplace learning (Kirby & Garrison, 1990). The faculty was uniquely equipped to provide the needed background in the theory and practice of adult and continuing education "and to provide it on a part-time basis for practitioners from a variety of disciplines who cannot leave their jobs to pursue full-time study" (Faculty Development Plan, 1990, p. 4). The Master of Continuing Education (MCE) was given final approval by the University of Calgary's General Faculties Council in the spring of 1994 after five years of development (Garrison & Kirby, 1995). It was the first graduate program developed and delivered by this faculty, as well as the first one offered primarily by computer mediated communication (CMC). As such, the MCE has been closely monitored by the University's senior administration, other academic faculties interested in developing a distanced-delivered graduate program, and Calgary's corporate community, the source of many of its students. The five-year critical appraisal that was recently undertaken was therefore carefully reviewed by these stakeholder groups.
The goal of the MCE program is to produce graduates who have a broad and critical perspective of the field of "workplace learning"; an appreciation for the linkages between theory and practice; a range of intervention skills to bring to organizations as workplace learning specialists; and an understanding of themselves as individual, team, and organizational learners. Although the program in this case study was delivered at the graduate level, this practically oriented goal is typical of professional development programs offered at most levels in adult continuing education units.
Three principles guide the design of the MCE program and define its delivery. The first principle is based upon the assumption that quality or depth of understanding is derived from critical discourse. Learning is organized around the discussion and critical analysis of content, not simply the dissemination of information. The second principle reflects the need for individuals to integrate new knowledge with previous understanding and experience, necessitating periods of private reflection and consideration of new ideas in terms of the individual's context. The third principle demands that each learner go beyond reflection and take the opportunity to apply the core content in a specific and practical situation (Proposal to Create MCE, 1994).

MCE Curriculum
The focus of the curriculum is adult learning and development issues within an organizational context. Through an applied approach, it facilitates the integration of organizational and adult education theory and practice, viewing organizational issues from a learning perspective. More specifically, the curriculum focuses upon administrative, programming, and facilitation activities of the professional who assumes a continuing education role within an organization or an association. The 12 half-courses that constitute the curriculum were carefully designed and structured to provide a meaningful sequence that is cumulative in terms of knowledge and skills acquired. Table 1 provides the theoretical framework that positions the courses with regard to conceptual and contextual factors. Students are encouraged to develop both specific capabilities and understandings from the various perspectives, ranging from the individual to a larger societal setting. Several themes (e.g., critical thinking, systems theory, ethics, and learning to learn) are emphasized throughout each course, and the program as a whole.

Structure and Delivery of the MCE Program
The educational experience is structured into large groups (face-to-face institutes), small groups (face-to-face institutes and CMC courses), and individual study (CMC courses). In terms of delivery, the goal is to build learning teams where students learn from and with other students. Although students may be geographically separated, communications technology allows them regular contact with their professors and learning group. Students gain access to the CMC courses via either their work or home computer stations.
The program is structured to be completed in a minimum of two years of part-time study. Students proceed through the program as a cohort of approximately 24, and take all of their core courses together. As a result, students typically complete their required courses in two years, before completing the final project, which may extend into a third year.
These course-delivery methods attract students from a wide geographical area. The first year of the program begins with an intensive face-to-face, three-week institute, followed by two core CMC courses delivered in the subsequent fall and winter terms. Teleconferenced (TC) classes may be held to supplement CMC during each term. Students work individually or in groups (using TC/CMC) on assigned projects between formal classes. The second year also begins with an intensive face-to-face, three-week institute, followed by two CMC courses delivered in the fall and winter terms. Towards the end of the second year (and frequently into the third), students complete their final project. The program culminates with an oral comprehensive exam (conducted face-to-face or by teleconference) to assess the student's grasp of theoretical and practical issues.

THE CRITICAL APPRAISAL MODEL
Stage One -where the interests and commitments of intended users of the evaluation are identified by bringing them together in some fashion to work with the evaluator and share in making major decisions about the evaluation.
In this stage, the evaluation committee members focus their discussion on clearly defining the evaluation's purpose and on how to use the results to improve the program. They collectively ensure that the evaluation process is clearly grounded in both adult education and organization development evaluation philosophy and practices.

Step 1: Identify Interests and Commitments of Potential Users
In August 1997, the Director of the MCE program appointed two members of the MCE Committee (made up of all full-time and adjunct MCE faculty) to a program evaluation subcommittee-the Critical Appraisal Committee (CAC). Its mandate was to conduct a critical appraisal of the MCE program. In September, an independent adult education consultant was added to this committee, and these three members thus became the evaluators of the MCE program, committed to working in close consultation with the MCE Committee members over the entire process.
The MCE Committee as a whole decided that the primary interest of the evaluation was to determine how well the MCE program was meeting its originally stated objectives, as well as the primary needs of its key stakeholder groups. In consultation with the MCE Committee, the CAC identified the key stakeholders as: MCE students and graduates; MCE faculty and administrator; original MCE program designers and their advisory committee; university support providers (MCE support staff, Distance Education Centre staff, library staff); the Faculty of Graduate Studies; and employers/clients of MCE students and graduates.
Step 2: Determine Primary Intended Users From the larger stakeholder group, the primary intended users of the MCE evaluation were identified as the MCE faculty and administrator. It was also noted that, as the Faculty of Graduate Studies would be reviewing the MCE program within the next 18-month period, the evaluation report should attempt to deal with as many of this particular stakeholder group's key questions/issues as possible.
Stage Two -where the evaluator and intended users commit to the intended uses of the evaluation and determine the focus of the evaluation, as the evaluator helps intended users formulate evaluation goals and expected uses.
In this stage, the committee makes sure that the evaluation data collection and analysis are clearly based on both adult education and organization development evaluation philosophy and practices. The committee also ensures that the data collected includes the potentially diverse perspectives of all past (i.e., original program developers) and present stakeholders in the program. In this case, that included MCE students and faculty, the Faculty of Graduate Studies, all service providers to the MCE unit and students, and the employers/clients of MCE students.

Step 3: Negotiate a Process to Involve Primary Intended Users in Making Evaluation Decisions
The CAC began the design process by soliciting feedback from the key stakeholders on all aspects of the MCE program relevant to each stakeholder group. Because this task involved an extensive time commitment, there was a lack of student or employer involvement with the CAC. Consequently, the data-gathering tools designed by the CAC attempted to compensate for this in the breadth and depth of data collected. The CAC also collected relevant existing data (including student course evaluations, records of services provided to the MCE unit, and evidence of employer support for students), along with new evaluation data (including student, faculty, service provider, and employer feedback on all aspects of the MCE program, and MCE original program designer and advisory committee feedback on current MCE program). CAC members did this using information sources and data-collection approaches that seemed most feasible given the time and resources available.

Step 4: Determine the Primary Purposes & Intended Uses of the Evaluation
The primary purposes as determined by the MCE Committee were threefold: first, to conduct an evaluation of the operation and outcomes of the MCE program using information from all stakeholders; second, to establish the future direction of the MCE program and the extent to which any modifications are required; third, to ensure that the MCE program documentation meets internal (Faculty of Graduate Studies) and external (potential partners in future MCE development and delivery around the world) requirements.
The findings of the critical appraisal would then be used to render a data-based judgement about what was working well, not working at all, or working but requiring alteration. These findings would also illuminate the program's strengths and weaknesses, the appropriateness of its targeted student audiences, its current effectiveness, and student/faculty/staff/ service providers' perceptions about the program. Finally, the findings would enhance the MCE Committee's knowledge base of successful distance graduate programs generally, thus increasing the MCE unit's ability to design and deliver future programs.
The CAC also recognized that the process of actively involving key stakeholders in issue clarification and analysis of results would have a number of secondary benefits. The MCE unit's and other key stakeholders' understanding about the critical appraisal results and how they would be used to guide the future direction of the MCE program would be enhanced. The involvement of all stakeholders now and in any future visioning processes would serve to support and reinforce the MCE program by increasing all MCE stakeholders' engagement in and sense of ownership of the program. Finally, the process would stimulate both the MCE program and the MCE unit's development as a team and vital member of the Faculty of Continuing Education and the University of Calgary academic community.
The MCE Committee decided there would be two reports written on the critical appraisal, to be known as Book One and Book Two. A description of the evaluation process, a summary of the data analysis, and recommendations to the MCE Committee were to be contained in Book One. Book Two would describe the MCE program's projected future operation and outcomes. Book One was to be written by the CAC chair upon completion of the data analysis; it would lay the foundation for Book Two, which would be written after the MCE Committee had had time to reflect on the evaluation findings. At that point, the committee could collectively create a blueprint/action plan for the next five years of the MCE program.

Step 5: Focus -Priorize Evaluation Questions or Issues
During the fall of 1997, the CAC met both as a subcommittee and with the entire MCE Committee to clarify the evaluation issues and any subsequent questions. In continual consultation with the MCE Committee, the CAC then designed a conceptual and procedural framework for data collection. First, overall program learning outcomes-derived from the MCE program philosophy, goals, and key principles-were defined for MCE students. Then, a number of outcomes were delineated, including teaching outcomes for MCE faculty in both face-to-face and on-line elements of the MCE program, employee professional and organizational development outcomes relevant to employers/clients of MCE students, and several MCE support service outcomes related to their contribution to the overall success of the program. Indicators for each of these outcomes were also specified. Finally, details of the data-collection process (the nature of the collection tools; the administration of each tool) were described, along with how results (i.e., data collected by each tool) would be analyzed. The analysis not only had to address the evaluation questions, but also how the evaluation findings would be shared with the MCE Committee to facilitate the collective creation of the final recommendations, and how they might be used to create a vision for the next five years of the program.
During the CAC's consultations with the MCE Committee, several political and power issues involving the key interests of each MCE stakeholder group were raised for discussion. Specifically, MCE students were very concerned that the program should be seen by both employers (for purposes of support and promotion) and other academic institutions (for purposes of pursuing further studies in the future) as credible. MCE faculty were concerned that the students positively evaluate their teaching/ supervision in light of the upcoming review by the Faculty of Graduate Studies so that the faculty could attract more corporate and institutional partners. Lastly, employers/clients of MCE students were concerned about real improvement in work performance as a result of MCE program participation.
The MCE Committee also acknowledged the ability of a credible, positive evaluation to increase the MCE program's marketability and its growth and development in desirable new directions. Given that two of the CAC members were MCE faculty, the danger that the critical appraisal would end up being merely a subjective endorsement of the current MCE program was seen as very real. Thus, the MCE Committee concluded that careful inclusion of feedback from all the key stakeholders would prevent "tunnel vision," by bringing to the evaluation a number of different perspectives and interests. The data-collection tools that were administered to the various stakeholders during December 1997 and early in 1998 were designed by the CAC (upon the approval of the MCE Committee) to determine the level of stakeholders' satisfaction with the following aspects of the MCE program.
1. Students: relevant administrative/operational aspects of the program; each course taken in the program (content and organization, quality of instruction, assignments/evaluation, teaching/learning environment experienced); the MCE program overall (effectiveness in meeting its stated objectives, quality of instruction, course delivery); effect on professional development and transfer of learning to workplace.
2. Faculty: the MCE program overall (effectiveness in meeting stated objectives, quality of instruction and delivery, students' academic performance); students subsequent professional development; adequacy of resources dedicated to the program.
3. Service Providers: face-to-face and on-line service provided to MCE students, faculty, and staff by the MCE librarian, the Distance Learning Centre (FirstClass system support), and the Faculty of Graduate Studies.
4. Support Staff: resources available to administer all elements of the MCE program; quality and quantity of communication with MCE administrators and faculty.

Original MCE Program Developers and Advisory Committee
Members: how well the current MCE program meets its original objectives and embodies the original philosophy and principles upon which it was based.
6. Student Employers/Clients: employees' (i.e., MCE students') application of newly acquired knowledge, skills, and attitudes in their workplace setting; employees' newly acquired professional capabilities and potential contributions to their organization.
Step 6: Simulate Use with Fabricated Potential Findings The various drafts of the design of each data-collection tool were critically reviewed by all CAC members to ensure that data collected would be useful in answering the questions outlined in Step 5. The inclusion of both closed/quantitative and open/qualitative questions in strategic spots in each tool was seen as crucial if respondents were to feel free to elaborate on answers to closed questions, as well as to add completely new ideas to the database. All of the structured interviews encouraged respondents to elaborate on their answers in order to fully express their particular perspective.
Stage Three -where evaluators and intended users make method, measurement, and design decisions after careful attention to issues of methodological appropriateness, believability of data, understandability, accuracy, balance, practicality, propriety, and cost.
In this stage, care is taken to obtain the perspectives of key past and present stakeholders in the program, in order to assess how well the program is meeting all of these potentially diverse needs. This entails the examination of the original purpose and assumptions upon which the program was developed, and critical reflection on whether or not the program's best interests are served by continuing with the original underlying framework. In this way, the committee is also laying the groundwork for the guidelines to be used for institutional critical selfreflection.

Step 7: Make Design Methods and Measurement Decisions
An overall conceptual framework was created and approved by all members of the MCE Committee before any work on the design of the evaluation strategy and tools was done. CAC members subsequently met several times to draft student and faculty questionnaires, as well as structured interview formats for collecting data from the MCE support staff, service providers, original program developers and advisory committee members, and employers/clients of MCE students. Although CAC members sought to ensure that as much credible and reliable data as possible were collected, they were also guided by the need for concise, easily deliverable/retrievable data-collection tools and the amount of time available for analysis of evaluation findings.
The evaluation strategy and measurement tool design was based on a number of assumptions about evaluation.
• Data needed to be both quantitative and objective (providing clear evidence upon which to base any program changes) and qualitative and subjective (allowing for the emergence of new knowledge about the program).
• Data needed to identify program elements that the MCE unit could then focus on for future planning purposes.
• Data-collection tools needed to have face validity and be as convenient as possible for all stakeholders to complete.
• Data-collection and analysis procedures needed to be consistent with the MCE unit's philosophy of program development, engagement of key MCE stakeholders as partners in the process, and corporate partnership building goals.
• The evaluation needed to focus on both the outcomes (important to the MCE unit, students, and students' employers/clients) and the process (becoming aware of the MCE program's strengths and weaknesses and subsequent changes needed) in order to be meaningful to all stakeholders.
In November 1997, the CAC met twice with the MCE Committee to priorize the evaluation questions and finalize the data-collection tools. These meetings were co-chaired by the chair of the CAC and MCE Committees, and decisions were made on a consensual basis. The CAC worked hard to ensure that the potentially diverse perspectives of the key stakeholders would be allowed to emerge in the data-collection process. CAC members continually revisited the initial stated purposes of the critical appraisal, keeping both the desired consultative nature of the exercise and the future program development goals in clear focus. All questions were weighted in terms of the value of the data obtained for improving the MCE program's goals, curricula, structure, and delivery to the most appropriate student audiences.
Step 8: Collect Data Evaluation data from the various stakeholders were collected from the following sources: 3. Service providers: existing formal course and institute evaluations from the beginning of the program; structured interviews conducted by CAC members in the winter of 1998.
4. MCE staff: face-to-face structured interviews conducted by a CAC member in the winter of 1998.

Original MCE program developers and Advisory
Committee members: structured telephone interviews conducted by a CAC member in the winter of 1998.
6. Employers/clients: structured telephone interviews conducted by both a CAC member and MCE Committee member in the winter and spring of 1998. All employer/client names were voluntarily provided by MCE students or graduates.
Step 9: Organize Data to be Understandable to Users In order to organize and present the data in the most accessible manner, the CAC obtained the services of the University's Office of Institutional Analysis. Staffed by specialists in statistical analysis and interpretation, the Office conducted frequency summaries of all quantitative questions on the student and faculty questionnaires, and advised the CAC on how to present the results so that those unfamiliar with statistical data could understand and interpret the resulting tables. All qualitative responses on these two questionnaires were summarized by one CAC member and then crossvalidated by the other two CAC members. MCE staff collected student profile information from the MCE files and did frequency counts and mean score calculations for all responses to the individual course and institute formal evaluations. All face-to-face and telephone interview data was recorded and summarized by the individual conducting the interviews. The chair of the CAC then took all of the data summaries and integrated them into Book One in draft form; this was presented to the MCE Committee for review and comment in the spring of 1998.
The following is a summary of the evaluation findings as they appear in the final draft of Book One, which integrated all MCE faculty feedback. MCE students were asked to respond to the draft version of Book One rather than the final version, due to timing (the last weeks of the winter academic term) and the limited resources available. Employer data was cross-validated as it was collected.
1. Students: appear to be experiencing a higher-quality intellectual and social environment than graduate students as a whole at the university; have a high level of satisfaction with the program as a whole, and especially with its focus on the application of theory to practice, the expert modeling of adult learning principles, and instructional techniques by MCE faculty; how the CMC component increases students' ability to access the MCE program and the ongoing resources available within it, the relevant and timely course materials that students use immediately in their own workplace settings, and the creation of a warm and inviting 'learning environment' that allowed for collaborative sharing, exploration, and learning.
Students' suggestions for improvements included: slow down the pace of both face-to-face and on-line courses; clarify instructor expectations; increase the variety and creativeness of instructional techniques; add a final large group (on-line) gathering of each cohort at the end of the program; build in more opportunities for students to both receive and give feedback; reduce the number of assignments (especially for on-line courses) while increasing the weight of marks given for on-line course participation; and increase instructors' skills in addressing individual learning styles and drawing students into on-line discussions. The most senior students offered some very thoughtful suggestions for improvements including: a foundational course/workshop in research and the use of technology-based teaching/learning strategies; the addition of electives on "Leadership," "Strategic Human Resource Management," and "Career Counselling." Overall, the MCE program appears to have consistently provided satisfying learning and personal growth experiences, while also providing some positive career enhancing outcomes. Comments also indicated that the MCE had significantly expanded the students' career prospects from the day that they registered in it, both directly (in terms of their increased visibility and credibility within their organizations and profession) and indirectly (in terms of their increased sense of confidence and possibilities). It appears that students have been able to apply their learnings from the MCE program immediately, indicating that it has proven to be very relevant and immediately useful in a number of diverse workplace settings.
2. Faculty: appeared to be reasonably satisfied with almost all aspects of the MCE program examined in this questionnaire, and especially with how well the program incorporates the three key principles upon which it was originally conceived. Their suggestions for improvements to the MCE program included: increasing resources; clarifying program objectives and student learning outcomes; critically examining course sequencing; allowing more flexibility in choice and order of core versus elective courses; developing new courses (research methods, writing, critical analysis of literature, business ethics, and HRM), and new concentrations (Workplace Leadership and HRM); improving faculty's skills in using on-line teaching/instructional technology; and exploring resource-saving partnerships/agreements with other graduate programs both in Canada and abroad.
3. Service Providers: the quality of service provided to MCE students, faculty, and staff by the contract library technician, Distance Learning Centre, and Faculty of Graduate Studies has been very high. Initial problems in the technical support of the FirstClass system were resolved and the current level of technical service appears to be meeting faculty and student needs.
4. Support Staff: appear to be satisfied with the amount of administrative support available to the program, their training in the use of FirstClass, and how effectively students are using FirstClass, but requested better communication with MCE faculty around issues that students frequently questioned the staff on.

Original MCE Program Developer and Advisory
Committee: appear to be generally satisfied with the current MCE program in terms of how they initially envisioned it. Overall, they would not change anything about the content or sequence of courses, but do have suggestions for how to modify some aspects of the program, which include: additional criteria for selection of adjunct faculty; alternative student supervision models; and more efficient ways to offer elective courses. The original members of the MCE Advisory Committee who were interviewed clearly thought that the current MCE program, while successful in its own right, was not fulfilling the original corporate and academic 'vision' of targeting senior corporate officers and more fully involving the business community in the delivery of the program.
6. Employers: appear to be quite satisfied with the professional outcomes achieved by employees in the MCE program, and would not hesitate to recommend the program to other employers, as well as support other employees who wished to enrol in it. They also had a number of suggestions for improvements to the program, including: change the name of program to more clearly reflect its focus in organizational learning; create an advisory committee from students and business partners to help shape the program; emphasize the development of a range of personal effectiveness skills that will assist students to be strategic thinkers and performance improvement consultants; and increase program flexibility by keeping core courses to a minimum.
In summary, the overall level of stakeholder satisfaction with the MCE program was very high. It was also clear from the data that, while the MCE program was meeting all stakeholders' key expectations, changes to make it even more responsive to some of these stakeholders were in order.
Stage Four -where intended users are actively involved in interpreting findings, making judgements based on data, and generating recommendations.
In this stage, the committee ensures that the interpretation of findings and recommendations are grounded in adult education and organization development evaluation philosophy and practices. The Chair facilitates the continued participation of all committee members, as well as other key stakeholders, in an ongoing process of program/organization development after the evaluation committee has disseminated its findings.

Step 10: Actively Involve Users in Interpreting Findings
In the fall of 1998, the CAC held a series of special meetings with the MCE Committee to receive feedback on all progressive drafts of Book One. After integrating this feedback into the final draft of Book One, CAC and MCE Committee members created 34 recommendations for the future direction of the MCE program that addressed the development needs that emerged. Co-chaired by the chairs of the CAC and MCE Committees, each meeting began with a synopsis of the previous meeting's decisions, after which specific objectives for the current meeting were stated and the floor was opened to comments and feedback from all members present.
These meetings resulted in many critical debates about all aspects of the MCE program, including its original philosophy, goals, and key principles, curriculum, structure and delivery, resources, and target student audiences. The debates included all members of the MCE faculty who chose to present their perspectives. In the process, it was decided that more data from employers/clients was needed in order to more fully represent this important point of view. A MCE Committee member subsequently conducted several structured telephone interviews with another 18 employers/clients whose names were volunteered by MCE students (see the "employer data" section of this paper).
During the development of the final set of 34 recommendations, MCE Committee members experienced an important transition-from an "old" to a "new" vision of the MCE program. This transition included becoming aware of how well the MCE was meeting its original objectives, exploring alternative MCE philosophies and subsequent structures by examining other comparable, successful programs across North America (see Massey-Hicks, 1999), and letting go of the "old" MCE in order to seriously hear and respond to changes being requested from student and employer/client stakeholders. It also included achieving an integrated view of a new MCE, one that would better meet student and employer/client needs, and starting to work collectively toward a new MCE model that would move the program into an exciting new developmental stage (see Brousseau, 1999).
The 34 recommendations were collectively drafted and presented by the Director of the MCE program to the Dean of the Faculty of Continuing Education in December 1998. These recommendations were subsequently presented to all members of the Faculty of Continuing Education, and made available to all stakeholders who participated in the data-collection process, prior to their final approval by Faculty Council in 1999.
Stage Five -where decisions about dissemination of the evaluation report are made beyond whatever initial commitments were made earlier in planning for intended use.
In this stage, the evaluation committee's decisions about the dissemination of the findings stem from critical self-reflection that is designed to create a guide to further action regarding the MCE program's ongoing improvement. This critical reflection exercise is grounded in adult education and organization evaluation philosophy and practice, and in a careful analysis of the current utility of the original purpose and assumptions upon which the MCE was originally developed.
Step 11: Facilitate Intended Use by Intended Users After the recommendations were approved by the Faculty Council, the MCE Committee held a number of half-day retreats to priorize them and then collectively create an action plan for implementing those of the highest priority for which resources were currently available. While these retreats were being held, members of the MCE Committee began to share the findings of the critical appraisal in presentations to colleagues at both local and international academic events. These presentations served to raise the profile and credibility of both the MCE program and the Faculty of Continuing Education at the University of Calgary, as a source of highquality graduate programs.
Step 12: Disseminate Findings to Potential Users In the spring of 1999, an "Executive Summary" of Book One was created and distributed to all stakeholder groups in the MCE program, along with an invitation to provide feedback to the Director of the MCE program on the critical appraisal's findings and recommendations.
All MCE Committee members continued to participate in various ways in further refining and implementing the MCE action plan. Some assisted the MCE Committee member responsible for writing Book Two (Bratton, 1999) in defining this document's content and direction. Others became involved in the formal MCE program review process undertaken by the Faculty of Graduate Studies in the fall of 1999 or revisited the individual courses they taught and students they supervised to see how each individual element of the MCE program contributes to the overall program objectives. Still others wrote/published scholarly articles on the development and delivery of distance continuing education programs for adults on workplace learning, or sought out potential partners to further develop and expand the scope of the MCE program worldwide.
Book Two, a comprehensive description of the MCE program's projected future for the next five-year period, was completed in December 1999. It details all of the changes made to the existing MCE program that arose from the critical appraisal process. Key amongst these changes are: • the creation of an additional concentration to the program that will allow students to specialize in either workplace learning or workplace leadership development; • the reduction of the core component of the program to give students increased flexibility in course selection; • the recruitment of additional faculty and staff; • the development of international agreements to deliver the MCE program abroad.
Overall, the critical appraisal participants agreed that, in addition to fulfilling its central evaluation mandate, the process had enhanced the MCE unit's and key stakeholders' understanding about the evaluation results and how they have guided the future direction of the MCE program. It also supported and reinforced the MCE program generally, increased all stakeholders' sense of ownership in the MCE program, and stimulated a sense of team and community within the MCE unit.

CONCLUSION
The Critical Appraisal Model of program evaluation that is described and illustrated in this paper is intended to contribute to the articulation of an approach to program evaluation that is accountable and responsive to stakeholders. In actual practice, adult continuing education units across Canada have successfully used such a proactive, learner-centred, and action research-oriented approach for decades, producing high-quality programs that meet the ongoing professional development needs of very diverse populations of adult learners.
Participating members of the Prairie Symposium on Research on University Continuing Education are planning to evaluate the usefulness of the Critical Appraisal Model by evaluating a number of non-credit programs in a wide variety of adult continuing education programs (Sloane-Seale, 1999). Thus, the model will be validated and enhanced in the coming months and years by those engaged in adult program development and evaluation on a daily basis across the country.
As well, the Faculty of Graduate Studies at the University of Calgary has expressed an interest in exploring the model's potential applicability to more traditional academic graduate programs (Weiser, personal communication, 1999). Given today's climate of increasing accountability and client-responsiveness in post-secondary education, a future-oriented and critically self-reflective model of program evaluation that could be successfully utilized at all levels of the post-secondary system is a timely contribution to the field.

The Critical Appraisal Model of Program Evaluation
Stage One -where the interests and commitments of intended users of the evaluation are identified by bringing them together in some fashion to work with the evaluator and share in making major decisions about the evaluation. The focus is on defining the evaluation purpose and on how results will be used to improve the program in a collaborative process grounded in adult and organization development principles and practices.
Step 1: Identify Interests and Commitments of Potential Users Step 2: Determine Primary Intended Users Stage Two -where the evaluator and intended users commit to the intended uses of the evaluation and determine the focus of the evaluation, as the evaluator helps intended users formulate evaluation goals and expected uses. The focus is on ensuring that data is collected from all key past and present stakeholders in a manner consistent with adult and organization development principles and practices.
Step 3: Negotiate a Process to Involve Primary Intended Users in Making Evaluation Decisions Step 4: Determine the Primary Purposes and Intended Uses of the Evaluation Step 5: Focus -Priorize Evaluation Questions or Issues Step 6: Simulate Use with Fabricated Potential Findings Stage Three -where evaluators and intended users make method, measurement, and design decisions after careful attention to issues of methodological appropriateness, believability of data, understandability, accuracy, balance, practicality, propriety, and cost. The focus is on assessing how well the program meets stakeholders needs and on whether or not their best interests are being served by the basic goals and assumptions upon which the program is currently based. Guidelines for institutional critical reflection are also created at this point.
Step 7: Make Design, Methods, and Measurement Decisions Step 8: Collect Data Step 9: Organize Data to be Understandable to Users Stage Four -where intended users are actively involved in interpreting findings, making judgements based on data, and generating recommendations, in a manner consistent with adult and organization development principles and practices. As well, every effort is made to ensure that all committee members continue to be engaged in program/ organization development after the findings are disseminated.
Step 10: Actively Involve Users in Interpreting Findings Stage Five -where decisions about dissemination of the evaluation report are made beyond whatever initial commitments were made earlier in planning for intended use. The focus is on creating a critically reflective guide for the program's ongoing improvement, which includes examining whether its original purposes and assumptions continue to be relevant to stakeholders.