An innovative approach to assessing professional skills learning outcomes: a UAE pilot study

Industry and academia around the world stress the importance of professional skills (also known as soft skills, generic skills, or transferable skills) so it is necessary to be able to assess students’ attainment of these skills. An innovative method was developed in the USA for assessment of these skills in an engineering program (Ater Kranov, Hauser, Olsen, & Girardeau, 2008); this method was based around student discussion of an open-ended, unresolved, discipline-related problem, held face-to-face and subsequently analyzed using a rubric. In the research project described here, the method was adapted for the United Arab Emirates by writing appropriate scenarios for computing students, by modifying the rubric and by running the discussion on an online discussion board. The primary aims were to determine the feasibility of adapting the method and to examine its suitability. The results of the study showed that the method can be adapted and employed very successfully with UAE students. This paper presents the method, its adaptation and implementation, and the results obtained. Introduction As part of the ongoing transformation from teaching to learning in higher education worldwide, learning outcomes (LO’s) and their assessment have become crucial elements around issues of pedagogy, accountability and employability (Association of American Colleges and Universities, 2013; Suskie, 2014). LO’s describe what students are expected to know and be able to do at the end of a course of study, and at program level they describe the expectations at graduation. In a LO’s-based system, education is focused on output and the competences attained by the learner. A degree is awarded as recognition of having achieved certain predefined LO’s rather than as proof of participation and successful completion of a program. A focus on LO’s is currently underway in the Arabian Gulf region, with stakeholders continuing to advance the priority of student learning and employability. Governments have recognized the strategic role that a highly skilled and educated workforce will play (United Arab Emirates Federal Government, 2010; General Secretariat for Development Planning Qatar, 2008) and as costs for education have far outstripped inflation and consumers (primarily students and their families) have become more educated and demanding, there have been increasing calls for evidence that what is being promised is being delivered. In a recent survey by the Association of American Colleges and Universities (AAC&U) (2013), 93% of US employers indicated that the ability to think critically, solve problems and communicate clearly was more important than a student’s disciplinary knowledge. Within the United Arab Emirates (UAE) and Gulf region, employers also value these abilities more than specific disciplinary knowledge (Arab Thought Foundation, 2013). Unfortunately, these are abilities in which students in the region, and the


Introduction
As part of the ongoing transformation from teaching to learning in higher education worldwide, learning outcomes (LO's) and their assessment have become crucial elements around issues of pedagogy, accountability and employability (Association of American Colleges and Universities, 2013; Suskie, 2014). LO's describe what students are expected to know and be able to do at the end of a course of study, and at program level they describe the expectations at graduation. In a LO's-based system, education is focused on output and the competences attained by the learner. A degree is awarded as recognition of having achieved certain predefined LO's rather than as proof of participation and successful completion of a program.
A focus on LO's is currently underway in the Arabian Gulf region, with stakeholders continuing to advance the priority of student learning and employability. Governments have recognized the strategic role that a highly skilled and educated workforce will play (United Arab Emirates Federal Government, 2010;General Secretariat for Development Planning Qatar, 2008) and as costs for education have far outstripped inflation and consumers (primarily students and their families) have become more educated and demanding, there have been increasing calls for evidence that what is being promised is being delivered.
In a recent survey by the Association of American Colleges and Universities (AAC&U) (2013), 93% of US employers indicated that the ability to think critically, solve problems and communicate clearly was more important than a student's disciplinary knowledge. Within the United Arab Emirates (UAE) and Gulf region, employers also value these abilities more than specific disciplinary knowledge (Arab Thought Foundation, 2013). Unfortunately, these are abilities in which students in the region, and the . An innovative approach to assessing professional skills learning outcomes: a UAE pilot study. Learning and Teaching in Higher Education: Gulf Perspectives,13(1). http://dx.doi.org/10.18538/lthe.v13.n1.213 page 2 UAE in particular, are poorly developed (United Nations Development Programme, 2012). These types of abilities and skills which graduates need in order to participate effectively in virtually every profession are known by various terms such as 'professional skills', 'soft skills', 'generic skills' or 'transferable skills'. In this paper we use the term professional skills as it is the term used by the Accreditation Board for Engineering and Technology (ABET) (ABET, 2010). ABET is the world leader in engineering accreditation and the professional skills espoused by ABET are central to this research study. ABET (2010) describes six professional skills related to (1) communication, (2) teamwork, (3) understanding ethics and professionalism, (4) understanding global and societal contexts, (5) lifelong learning, and (6) knowledge of contemporary issues. The challenge surrounding these professional skills is that employers prioritize them, regionally students are weak in them, and they are considered difficult to assess (Shuman, Besterfield-Sacre, & McGourty, 2005). One of the main difficulties in assessing these skills is that while it is easy to understand what they are, educators struggle to define them clearly and to write rubrics for their assessment.
In order to evaluate the attainment of LO's in the professional skills it is of utmost importance to have reliable methods to conduct the assessments. This paper describes a pilot project conducted in a computing college to determine the feasibility of adapting an innovative American-designed approach to program level assessment of professional skills in a Gulf context. The approach, developed by Ater Kranov is centered on student discussion of an open-ended unresolved engineering related problem (Ater Kranov, Hauser, Olsen & Girardeau, 2008). In Ater Kranov's approach the discussions are held face to face, recorded and subsequently analysed using a rubric to assess the students' abilities in the professional skills. In the pilot project presented here the discussion is based on an authentic openended information technology related problem and the discussion is conducted via an online discussion board.

Context
This study was conducted at a UAE public university serving Emirati students in a gender segregated environment with campuses in Abu Dhabi and Dubai. Although the medium of instruction is English, nearly all students are native speakers of Arabic. The vast majority of students are first generation university attendees who have come from a developing K-12 system where achievement on international examinations has been historically poor. For example, in the 2012 Programme for International Student Assessment (PISA), UAE 15-year-olds consistently ranked in the bottom third internationally for mathematics, reading and science, while they were ranked 40th out of 44 in the problem-solving domain (Organisation for Economic Co-operation and Development, 2014). Similarly, the Arab Knowledge Report 2010-11 (United Nations Development Programme, 2012) reported that only 8.2% of UAE students achieved an acceptable level in terms of cognitive skills, i.e. searching and processing information, written communication, problem-solving, and using technology. This clearly indicates that a high percentage of UAE secondary graduates would have difficulty participating meaningfully in a knowledge-based economy.
The University at which the study was conducted has an institutional set of professional skills LO's that have been in place for over 10 years. To assure quality, the university maintains a well-established assessment program centered on these LO's. The University is internationally accredited by the Middle States Commission on Higher Education (MSCHE) and has disciplinary accreditations from a number of international accreditors including ABET. . An innovative approach to assessing professional skills learning outcomes: a UAE pilot study. Learning and Teaching in Higher Education: Gulf Perspectives,13(1 Good LO's should comprise succinct descriptions of the knowledge and skills arising from a meaningful learning experience. They are an attempt to make tangible what was once thought to be intangible, focusing on what is to be learnt, rather than taught. The University has six core generic skills LO's which form the foundations of all academic programs and which are the focus in program assessment. These University Learning Outcomes (ULO's) state that graduates will have adequate abilities in the following areas:  Language: Communicate effectively in English and Modern Standard Arabic, using the academic and professional conventions of these languages appropriately.  Critical Thinking and Quantitative Reasoning: Use both critical and quantitative processes to solve problems and to develop informed opinions.  Global Awareness: Understand and value their own culture and other cultures, perceiving and reacting to differences from an informed and socially responsible point of view.  Information Literacy: Find, evaluate and use appropriate information from multiple sources to respond to a variety of needs.  Technological Literacy: Effectively understand, use and evaluate technology both ethically and securely in an evolving global society.  Leadership: Undertake leadership roles and responsibilities, interacting effectively with others to accomplish shared goals.
The American-designed approach which is being followed in this study was developed for assessment of ABET's professional skills. Within ABET's Computing Division, there are 14 LO's, relating to skills, knowledge and behaviors that describe what students should know and be able to do by the time they graduate. Among these are professional skills outcomes, and four main ones closely related to the institution's ULO's were targeted in this study. These are:  Ability to function effectively on teams to accomplish a common goal (ULO-Leadership);  Understanding of professional, ethical, legal, security and social issues and responsibilities (ULO-Global Awareness and Critical Thinking and Quantitative Reasoning);  Ability to communicate effectively with a range of audiences (ULO-Language);  Ability to analyze the local and global impact of computing on individuals, organization and society (ULO's-Global Awareness and Critical Thinking and Quantitative Reasoning) (ABET, 2010, p. 3).
The University follows LO's assessment processes which are recommended by leading assessment scholars and accepted by the major accreditors. Walvoord (2010), for example, has a set of guidelines on how to assess LO's at the program level. She describes how faculty should select an appropriate method of assessment by ensuring alignment between the LO and method, conduct the assessment, analyze the results, then meet to discuss the results and bring forth any suggested changes if the performance of the students did not meet expectations. Though course embedded assessments are the most common choice, the focus of assessment, for both Walvoord (2010) and our institution, is at the program level, not at the level of the individual student. There are two broad categories of methods for conducting assessment: indirect and direct (Banta & Palomba, 2015;Suskie, 2009;Walvoord, 2010). Indirect methods are proxy indicators of student learning such as student ratings of their own knowledge and skills, end of course evaluations about the course or employer satisfaction with graduates. Direct methods are tangible and compelling examples of student learning such as samples of student written work, capstone projects or responses to an examination (Suskie, 2009 accurate portrait of student achievement, triangulation (i.e. using multiple sources of evidence) to evaluate student learning helps to get closer to the truth (Rogers, 2010).

Literature review
In recent years, stories of student learning in higher education, or lack thereof, have inundated the popular press and higher education publications. The main publication leading this barrage was Academically Adrift: Limited Learning on College Campuses (Arum & Roksa, 2010), a book which reported on student results in the Collegiate Learning Assessment (CLA). The CLA is a standardized test from the US which utilizes an open-ended scenario-based performance task methodology and purports to assess problem-solving, critical thinking and written communication. Arum and Roksa (2010) found that 45% of college students did not demonstrate any significant learning in these domains over their first two years of college. As important, but lacking the mainstream publicity, the Wabash National Study in the US reported similar findings (Blaich & Wise, 2011). The Wabash study used different assessments which included the standardized Collegiate Assessment of Academic Proficiency and the Miville-Guzman Universality/Diversity Scale (Pascarella & Blaich, 2013). The results of both these studies showed that a considerable percentage of students were not learning as much as expected or desired. In Arab countries this weakness in learning is also a serious concern: McKinsey (2011) reported that only 20-35% of employers in the Arab world felt that university graduates had the skills needed to be successful in the workplace.
While most educational institutions have some form of quality assurance program to maintain standards, these recent studies highlight the importance of assessment of academic programs in order to identify deficiencies and improve quality. Much of the recent emphasis on continuous improvement of programs is driven by accreditors such as the MSCHE, an institutional accreditor, or ABET, a disciplinary accreditor. The processes expected by accreditors of institutions to improve program quality through improved student learning are often quite similar and may be summarized as in ABET documentation: Can the program demonstrate the level to which students have attained the anticipated student outcomes? [...] The evidence of student learning is then used to identify student strengths and weaknesses related to each of the student outcomes for the purpose of making decisions about how to improve the program teaching/learning processes. (ABET, n.d., Assessment and Evaluation of Student Outcomes Section, para. 1) The strong focus across accreditation bodies on student attainment of LO's is very clear. For example, within MSCHE from 2009 to 2013, assessment of student learning was included in 58% of all warnings or probationary actions taken by the Commission against member institutions (Klinman & Fogarty, 2013) In 2006 ABET commissioned a study into its newly adopted accreditation criteria, which included the professional skills; they found that 2004 graduates were better prepared than 1994 graduates because of the new emphasis on student outcomes and the professional skills (Lattuca, Terenzini, & Volkwein, 2006). Professional LO's are particularly valued by various stakeholders. Passow (2012) surveyed engineering graduates of a large university to determine which LO's they deemed to be most important for their working lives. The graduates rated the professional skills of teamwork, communication, and problem solving amongst the most essential, over many of the more technical LO's. developing students' technical LO's, but that teaching and assessing the professional skills are more challenging. To this day, authors continually support this assertion through statements such as "professional skills are not easily incorporated in traditional engineering classes" (Barakat & Plouff, 2014, p. 513). In the Arabian Gulf region, AlBahi, Taha and Turkmen (2013) confirmed that similar difficulties exist. Nevertheless as key ABET student outcomes, the professional skills are expected to form a key part of an engineering or computing curriculum.
The challenges that assessment of the professional skills have posed has led faculty members to attempt to use a number of different assessment methods. McNair, Paretti, Knott and Wolfe (2006) used an eportfolio in first-year courses as a way to allow students to demonstrate attainment of professional skills and for faculty to assess them using developmental rubrics. Two of their findings were that faculty felt uncomfortable with assessing the professional skills and struggled to integrate the rubrics into existing class work. Barakat and Plouff (2014) developed and implemented an online system to teach and assess the professional skills during the student internship. They found that while their approach was useful, there were many challenges to assessing professional skills. On a more traditional note, Lopez, Cruz, Sanchez and Fernandez (2011) utilized an open-ended, take-home, written exam to assess the professional skills. While they had success with this method, there are clear limitations in its ability to assess the professional skill of teamwork. Internships evaluated by rubrics have also been used and, while this allows assessment of teamwork, there is little control over the trustworthiness of the assessment results because the employer completes the rubric (AlBahi, Taha & Turkmen, 2013). An indirect method of assessment by Richerson, McAteer, Spencer and Scheibler (2007) utilized a portfolio in which students identified instances where they demonstrated the professional skills. They then conducted a self-assessment to identify strengths and weaknesses, and developed strategies to improve. The role of faculty was to ensure student engagement and to assist them in setting realistic goals. While self-assessment is a useful skill, it does have weaknesses in terms of reliability as a method because it is an indirect measure.
In 2007, Engineering faculty at Washington State University, led by Ater Kranov, collaborated to develop an authentic performance task with an accompanying marking rubric to assess ABET professional skills outcomes (Ater Kranov, Hauser, Olsen & Girardeau, 2008). Small groups of randomly selected senior students were presented with a real-world, unresolved engineering problem and asked to discuss and propose approaches to the complex issue. Students were given 45 minutes to discuss the scenario and possible solutions; a facilitator was present but provided prompts only if a group was off track for 15 minutes. The discussions were recorded and, subsequently, groups of faculty examined the transcripts using the professional skills rubric. A norming process was undertaken to improve inter-rater reliability and discussions with rationales were used to develop a level of inter-rater consistency. The rubrics were scored on a six-point scale and faculty determined that a score of four represented the target for graduating students. Scores were averaged for each professional skill and reported as a single score. The researchers concluded that their method was an appropriate and valuable way of assessing professional skills LO's. Since the initial publication by Ater Kranov et al. (2008) forums to facilitate the demonstration of learning and skills has a rich theoretical and empirical foundation. From the seminal work of Garrison, Anderson, and Archer (2000), the Communities of Inquiry Model has indicated that online discussion forums are able to foster discourse and reflection in a manner superior to more spontaneous face-to-face classroom discussion. Salmon (2011) showed that co-constructing knowledge and working effectively as a group, which are important aspects of learning, develop quite strongly in an online environment. Within a non-native language learning environment, time for reflection is thought to be especially critical for students to respond to the best of their ability. Anxious or nervous non-native language learners are thought to possess a powerful 'affective filter' (Krashen, 1982) which can negatively impact their communication and learning in spontaneous face-toface environments.

Research Questions
From the research to date it is known that the method of Ater Kranov is a valid and reliable method for assessing all of ABET's professional skills among engineering students in an English language environment. This led to the current research project where the researchers examined whether it was feasible to adapt the method to a UAE environment. In particular it led to the development of the following research questions: 1. Is the implementation of this innovative approach to the assessment of professional skills LO's effective in the UAE with non-native speakers of English? 2. Is the implementation of this innovative approach to the assessment of professional skills LO's effective if conducted online rather than face-to-face? 3. What, if any, adaptations to the approach could improve this method of assessment?

Methodology
In order to conduct this assessment with a group of students the most feasible approach was to select a course and include this assessment as a mandatory item. As this was a pilot study, only one course was chosen, as it provided an adequate number of students. The selected course within the computing college was a third year requirement which investigates the role of information technology in global and local contexts. The course LO's (linked to ULO's) included Global Awareness, Technical Communication, Leadership (including teamwork) and Critical Thinking, which made this an ideal course to assess the ABET professional skills representing teamwork, global perspective, ethical and social awareness and communication. Approval to conduct the study was obtained from the University Ethics Committee.
The participants were 27 native Arabic-speaking females, in their early 20's, who had been studying in an English-medium university throughout their baccalaureate degree. The English language entrance requirement for the institution is an International English Language Testing Service (IELTS) level of 5.0, which corresponds with modest users of the language (IELTS, n.d.); many students graduate at IELTS 6.0, which corresponds with competent users (IELTS, n.d.). As the participants were third year students, their IELTS level could be estimated on average at around 5.6.
For this study we took four major steps in adapting and implementing the original method: 1. Scenario adaptation and development; 2. Rubric adaptation; 3. Scaffolding the three-phase discussion forum; and 4. Rubric assessment and establishing of inter-rater reliability. Schoepp

Instruments
In the first step, adapting existing scenarios and developing others, particular attention was paid to English language readability. A faculty member in the University's foundational English language program was contracted to undertake this adaptation and development. Three scenarios were completed and each was kept to around 600 words and maintained a Flesch-Kincaid readability level of 12 (roughly equivalent to average school grade 12 reading level). The scenarios were real-life openended interdisciplinary problems related to computing that were to be discussed in small groups. Of the three scenarios, two were adapted from previous work (Ater Kranov et al., 2008) and made more linguistically and contextually appropriate, while the third, titled "Who controls the web?" (Appendix A) was written in-house. The accompanying set of guiding questions was also developed at this time.
As the scenarios were being created, the rubric used for evaluation of discussions was also being adapted. The rubric was developed through an iterative process in which minor modifications to the rubric of Ater Kranov et al (2008) were made and debated between the researchers until consensus was achieved. An example of an alteration to the rubric was that language accuracy was added as a criteria because of the second language context. In addition, the definition of the outcome may also have been edited to better focus the instrument. For example, Appendix B's, definition for Outcome F clearly refers to both language accuracy and communication with stakeholders. The aim of the rubric development was to have a tool which would, in a valid and reliable manner, facilitate trustworthy scoring of the student discussions. To achieve this, the following three questions suggested by Moskal and Leydens (2000) guided the process: 1) Are the scoring categories well defined?
2) Are the differences between the score categories clear? and 3) Would two independent raters arrive at the same score for a given response based on the scoring rubric? (para. 23) In the final iteration, the rubric comprised a seven-point scale: 0-2 Beginning, 3-4 Developing, and 5-6 Accomplished (Appendix B). A score of 4-5, straddling Developing and Accomplished levels, was the hoped for target given that students were in Year 3 and that the exercise was towards the latter part of the academic year. Nevertheless, this first round of rubric implementation was also expected to serve as a benchmarking exercise.

Procedure
The third step was the actual roll out and implementation of the scenario-based discussions. A graded progression of scaffolding (Wood, Bruner & Ross, 1976) was implemented, in which students participated in three discussions: one face-to-face; one online and formatively assessed; and one online and summatively assessed. The 27 students were randomly assigned to five discussion groups for each phase of the study.
Discussion 1 -Face-to-face: Students were provided with a set of instructions, a scenario with question prompts and the rubric against which they would eventually be assessed. Specific prompts or guiding questions were used to steer the discussion because Ater Kranov (2013) had previously found that prompts improved student performance on the task. After reading the scenario, students began to address the questions, and at times came to consensus or diverged in different directions. The teacherresearchers circulated amongst the groups helping to facilitate the discussions and answer technical or procedural questions. To complete the task, a whole class dialogue around the scenario, discussion, and rubric was led by researchers. . An innovative approach to assessing professional skills learning outcomes: a UAE pilot study. Learning and Teaching in Higher Education: Gulf Perspectives,13(1 Discussion 2 -Asynchronous online: After a brief review of discussion forum functionality, online forum etiquette and discussion expectations, students were given two weeks to repeat the process using a different scenario. The researchers mostly remained as online observers, but occasionally intervened if a student dominated a discussion or did not participate. Solutions to the issues posed in the scenarios originated from the students, never the researchers. To conclude this second phase, students were debriefed and provided with examples of discussion posts which had informally benchmarked across the rubric spectrum.
Discussion 3 -Asynchronous online: This final discussion mirrored the process followed in Discussion 2. Groups were allowed two weeks to discuss a new scenario and respond to the questions. Students were aware that their groups were being summatively assessed according to the rubric. To maintain motivation, students were also assessed individually according to an additional set of criteria which included level of contribution, sophistication of comments, ability to work with others and English language proficiency. A simple rating scale was used for this purpose.
Working in isolation, each of the two researchers scored Discussion 3 for the five groups, making notes and highlighting the transcripts. They also provided a rationale for their scores. Coming together scores were then compared and discrepancies were discussed and debated until consensus was achieved. The process involved sharing of rationales, re-examining transcripts and awarding half marks, and was built on Stemler's (2O04) concept of the consensus estimate where reasonable raters should be able to come to agreement on a good rubric. For each of the four LO's, each group received a score between 0-6, with 4 being the minimum score hoped for.

Results
The scores achieved by each group for each outcome along with the mean scores are shown in Table 1. As this exercise concerned the assessment of LO's at the program level the data, at its most aggregated level, were of the most consequence. Individual student results were not a consideration. The LO where the students performed best was Teamwork, with a mean score of 4 (4, straddling the Developing and Accomplished levels, was the target), while Ethical and Social Awareness had the poorest performance with a mean score of 3.3. In addition, both Communication and Global Perspective had means below 4. Another consideration was the performance of the individual groups themselves. The potential benefit of disaggregating the data to this level is that if one group performed particularly high or low, the existing analyses and results may need to be reconsidered. Only Group 3 achieved an overall mean score of 4. Group 4 had an especially low mean score of 2.8 with no LO scoring at least 4. Groups 1, 2, and 5 had mean scores ranging from 3.5-3.9, indicating near attainment of 4.
The results are also shown graphically in Figure 1, where the number of groups ≥ the target of 4.0 is presented 1 . Four of the five groups scored at least 4 for Teamwork, while for both the Global Perspective (1 group) and Ethical and Social Awareness (2 groups) LO's less than half of the groups achieved the target. From the overall program standpoint, this data confirms the findings presented from a mean score perspective. Teamwork is clearly the LO where the students performed best and the only one on target, while Ethical and Social Awareness along with Global Perspective were the LO's where students demonstrated a significant weakness.

Discussion
As a main aim of the research was to investigate the suitability of Ater Kranov's method for use in the UAE, we were primarily concerned that the assessment produced valid and reliable results. The results should give us a reasonably good indication of the students' ability levels in the professional skills, and they appear to have done that.

Our first research question was: Is the implementation of this innovative approach to the assessment of professional skills LO's effective in the UAE with non-native speakers of English?
The answer is that it was, without doubt, effective. The scenario-based online discussion forum proved to be a success and the results demonstrated clearly students' strengths and weaknesses. With regard to the English medium environment presenting difficulty, the Communication LO, though falling just short of our target, had higher scores than either the Ethical and Social Awareness or Global Perspective LO's. The fact that all students were non-native speakers of English with moderate levels of proficiency led us to believe that effective written communication would pose a challenge. This did not appear to be the case. It may be that the reflective time permitted as part of asynchronous discussion forums (Garrison, et al. 2000) was ideally suited to non-native speakers because editing before posting was possible.
Hence, the answer to research question 2, is the implementation of this innovative approach to the assessment of professional skills LO's effective if conducted online rather than face-to-face? is also positive.
Our third research question was: what, if any, further adaptations to the approach would improve this method of assessment? We identified a number of ways to improve this assessment method. Firstly, it was felt that the rubric needed further refinement to increase understanding between researchers and make it better aligned to our particular context. For example, the Communication section of the rubric should be further altered to consider the second language nature of the students and focus more on accuracy. This may entail greater detail in terms of grammatical precision, vocabulary and spelling. It was also felt that the Ethical and Social Awareness and the Global Perspectives rubric sections were too similar and should be re-examined. Secondly, to make the rubric more applicable to other disciplines, it could be re-framed to better align with the institutional ULOs rather than with the ABET professional skills. This would still permit rubric use within the computing college, through mapping of their program LO's to the ABET professional skills, but it would allow other colleges to use it as well. Thirdly, the discussion questions or prompts should be further refined. In order for the students to effectively focus on the LO's assessed by the rubric, the questions should better align with the LO's. However, it is important that the prompts do not become too restrictive as the open-ended nature of the task is essential for meaningful discussion and assessment. Also it is important that there is no (or minimal) teacher-researcher involvement in the discussion forums so as to avoid falsely enhancing student performance.

Conclusions
Overall, the scenario-based online discussion forum proved to be an effective method of assessing professional skills LO's since the open-ended nature of the discussion meant that students were required to engage in an intellectually meaningful manner. While the innovative nature of a discussion forum as a method of LO's assessment, proved engaging for the students, it was a labour intensive process for the teacher-researchers. However, the valuable contribution that this pilot has made to assessing LO's within the College has meant that further implementations are planned. Results from this method will be used as part of our Assess-Analyze-Act model of LO's assessment, and will feed directly into evidence for program accreditation. Another major benefit of this pilot is that it has shown that even though this method was designed for the field of engineering using ABET's professional skills, there is potential for it to be applied to other disciplines because of the cross-disciplinary nature of these LO's.

College of Technological Innovation-Professional Skills Evaluation
The following ABET learning outcomes, which are linked to the MALOs, state that you should have the: d) ability to function effectively on teams to accomplish a common goal e) understanding of professional, ethical, legal, security, and social issues and responsibilities f) ability to communicate effectively with a range of audiences g) ability to analyze the local and global impact of computing on individuals, organizations, and society The following activity is designed to elicit your knowledge and ability to apply professional skills. Its purpose is to determine how well your IT program has taught you these skills. By participating in this online discussion, you are giving your consent for the research team to use your anonymous posts for academic research purposes.

Discussion Instructions
Imagine that you are on an interdisciplinary team working together for a government organization on the issues raised in the scenario.

Who controls the Web?
In 2012 there was a United Nations-International Telecommunication Union (ITU) conference in Dubai where 193 countries looked at changing some of the global telecommunication rules that exist in an international treaty. Most of these rules had not changed since 1988, before the introduction of the internet. Because of this, the ITU was looking at updating some of the rules to fit today's internet environment. The conference was supposed to discuss technical issues but instead became a discussion on freedom of speech and the openness of the internet. During the talks at the conference over possible internet rules, it became clear that there were two sides developing.
There was a Western group that wanted to make sure there were no United Nations rules on cyberspace, saying they could damage Web business and allow governments to put limits and checks on the internet. This group argued that the new United Nations agreement should simply not mention the internet at all because it is a service that runs on top of telecoms systems. There was also a second group, including China, Russia and some Gulf States, that wanted the United Nations to set up rules for stronger control by governments over all levels of internet. These countries said the internet should be part of the agreement because it uses telecoms.
The US was first to declare its opposition to the draft treaty. "The internet has given the world great economic and social benefit during these past 24 years. All without UN regulation", the leader of the delegation said. Internet engineers, including from big companies like Google and Microsoft, as well as academics are worried that changes to the UN telecommunications treaty will give governments more control of the Internet in their countries. They argue that any United Nations rules for greater government control on the net could be used for even more control from Web watchers in places such China, Iran and other nations. Therefore the United States, along with the United Kingdom and Canada, refused to sign the United Nations treaty on telecommunications and the Internet. They have indicated that they instead want private companies to drive internet standards. "Internet policy should not be determined by member states but by citizens, communities, and broader society" Terry Kramer, the US ambassador said.
The US was joined in its opposition by 55 other countries. All said they would not sign the proposed final text, meaning that although a number of other countries will sign it, the treaty cannot be implemented.
However, according to Mr Al Ghanem, who is director general of the UAE's Telecommunications Regulatory Authority, there was a massive campaign against the conference. He said some people claimed "that this conference would take over the internet and change the landscape of the internet" before they knew what would be in the agreement. He claimed the treaty recognized that administrations do have access to international telecoms services. "The treaty says member states shall affirm their commitment to human-rights obligations". According to Mr Al Ghanem and the other delegates that wanted to include internet in the treaty, the issue was a technical one and the treaty does not address content. "This is a technical agreement, and not about human rights or freedom of speech," he said.
The failure to reach an agreement in Dubai could mean that there will be regional differences in internet use. "Maybe in the future we could come to a fragmented internet," Andrey Mukhanov, of Russia's Ministry of Telecom and Mass Communications, told the Reuters news agency. "That would be negative for all".

Group Number_________________________________
Note: A score of 4 represents competency for students in CIT 305.
Outcome (d). Ability to function effectively on teams to accomplish a common goal Students work as a team to address the problem by acknowledging and building on each other's ideas. Students collaboratively build an understanding of the issues involved and possible approaches to the problem. Students clearly frame the problem or issue and begin the process of resolution.
Beginning Developing Accomplished 0 1 2 3 4 5 6 *Students do not acknowledge other student's ideas. A few students may monopolize and make decisions for the whole team. *Interruptions and cross-talk are frequent. *Students occasionally acknowledge and build on other students' ideas. *Students occasionally ask for ideas from students who are not sharing.
*Students acknowledge and build on the positions of other students. *Students clarify each other's ideas and encourage participation.
*Students do not identify or summarize the problem.
*Students start to frame the problem, although some key details are glossed over. *Students discuss one or more approaches to resolve the problem.
*Students clearly frame the professional challenge and issues related to the problem. *Students develop appropriate, concrete approaches to resolve the problem.

Outcome (g). Ability to analyze the local and global impact of computing on individuals, organizations, and society
Students consider how various aspects of or approaches to the problem affect individuals, organizations, and society in local and global contexts.

Beginning
Developing Accomplished 0 1 2 3 4 5 6 *Students do not make connections between computing and other contexts. *Students do not consider the impact of computing problems and solutions on individuals, organizations, or society.
*Students briefly consider some impacts of computing problems and solutions on individuals, organizations, and/or society.
*Students analyze the local and global impacts of computing on individuals, organizations, and/or society as appropriate. *Students integrate these considerations into their approach(es) to the problem.

Comments:
Scoring Instructions: Scoring: Estimate a score for team performance on each dimension using the evidence that you gathered from the discussion. Where there is more than 1, add together and divide by 2, then round to the nearest whole number. Write your score for each group in the chart below.
Comments: Record your rationale for your final scores. (For example, "The team addressed skill 3e, but did this superficially, so I gave them a 3 overall on this," or, "Student 7 addressed skill 3g thoroughly, but the team didn't acknowledge his ideas or integrate them into their understanding, so I gave the team a 2 on this," etc.)

CAC Skills Score
3d. Ability to function effectively on teams to accomplish a common goal 3e. Understanding of professional, ethical, legal, security, and social issues and responsibilities 3f. Ability to communicate effectively with a range of audiences 3g. Ability to analyze the local and global impact of computing on individuals, organizations and society TOTAL