An Evaluation of CALL Software : UUEG

This study attempts to evaluate the CALL software package, 'Using and Understanding English Grammar' (UUEG). This evaluation will be carried out using the judgmental perspective suggested by Chapelle (2001). The revolution of technology has started to invade classrooms and as the hard copy of the book is widely used in Saudi Arabia (including the author) an evaluation of its soft copy will inform us of the usefulness (or otherwise) of this software. The application of the software in the author’s context did not have much of great influence in students’ achievement; the total scores of students, when analyzed by t-test, were significantly different though. It also affected their learning motivation, excitement and attendance. However, it must be taken into consideration that pedagogical objectives, learners’ differences, learning styles and strategies and teachers’ own styles and expertise all vary from context to context and accordingly the value of all CALL software will be context dependent any may not necessarily lead to success.


INTRODUCTION
The importance of CALL evaluation can be likened to the parts of a jigsaw, as each factor contributes and leads to another; indeed, Sheldon (1988) notes that the selection of a course book is an 'educational decision' that involves 'considerable professional, financial and even political investment' (p.237).In the author's opinion, this is also true of CALL-especially when its application involves a lot of parties.
The fact that the author teaches from the book that the software is based on is one of his reasons for doing this evaluation.It allowed him to distinguish any learning differences between the group who is using the software and the groups who used the original book in previous years.Furthermore, the author hoped that this evaluation might allow to form a positive conclusion on the benefits of teaching using CALL, which the author could use to persuade the author's administrators of the advantages of using this software.At present, they are not as enthusiastic about it as the author himself.
Having taken these reasons into consideration, we can begin to see parts of the CALL evaluation jigsaw emerging.A further factor that adds to the puzzle is one which is every teacher's demand: we need to determine whether or not this software has the potential to maximize the optimal conditions for language acquisition, as there are hundreds of CALL 'materials which claim to be leading-edge' (Reeder et al., 2004).Nonetheless, previous studies on the subject-e.g., Oates (1981), Higgins (1985), Askildson (2011) and Hwa (2012) -have proven the teaching of grammar through CALL to be effective and most beneficial.The current study should highlight the influence of CALL on low proficient learners which has not been a popular topic in the field of CALL, as most of previous studies focus on a comparison between different groups.

MATERIALS AND METHODS
The participants: They were 35 male freshmen students majoring in English who were repeaters of a grammar course.They took the course a semester before this study with the same materials except they studied the hardcopy of the book and in this study they were presented with the software.All teaching took place in a lab for a 14 solid teaching weeks.It was the expectation that introducing this new treatment would help to improve the student's knowledge in grammar and hence increase their achievement and their chances to pass the course.Learners were familiar with computers and some training took place before the actual data collection commenced.
The software: Before beginning the evaluation itself, it is necessary to give a brief description of the software, which is based on Azar (2009).Due to space restriction, the author will only provide an analysis of just one chapter of the book with intercepted description of the methods used in implementing the software in classroom.The analyzed chapter is divided into four parts, each focusing on the following tenses: the present perfect, the present perfect progressive, the past perfect and the past perfect progressive.Each section includes several quizzes, exercises and one crossword game and these are followed by three main tasks covering listening, speaking and reading comprehension (named by myself).To finish, there Analytically speaking, the chapter follows Ur (1988) for teaching grammar: presentation, explanation, practice and test.The chapter starts with a preview of the tense, comparing it to and/or contrasting it with, similar tenses-a method that is claimed to be effective by Walker (1967).Learners can either read or listen to the preview before examining a chart that exemplifies the tense.Following this, students are presented with a range of nearly all the typical mechanical drills, such as gap filling, error recognition, cloze and multiple choices.Some of the quizzes come with animated pictures and the exercises are represented in a linear progression-i.e., they become more difficult as the students advance.The author would consider some of these exercises to be preparatory activities for the main tasks; for example, exercise 11 (Fig. 1) prepares the students for the speaking task in exercise 16 (Fig. 2).
Within the program there are five main buttons located at the top of every page.These are made up of 'outline' (which outlines the whole chapter in detail), 'report' (enabling students to check their progress after each step), 'glossary', 'help' (where learners find help topics) and 'contents'.The tasks: The listening task suggests that students listen to the recording of an international student's experience before answering the corresponding questions.A transcript of the dialogue is available.
In the speaking task (Fig. 2) there is a 'record and compare' function that enables learners to listen to a prompt before reiterating the sentences whilst recording their speech.This enables them to compare their recordings to those of the model.Transcripts of the prompts and the model's words are available and it is possible to play both of the recordings again and again.
The reading task comes in the form of a passage that includes some difficult hyperlinked words.By clicking on each, there appears a pop-up window that is linked to the glossary page.This displays the word's meaning along with a list of the other hyperlinked words, thus allowing students to check the meaning of other vocabulary.Multiple-choice comprehension questions follow the passage.
The above outlines what the software suggests for each task.However, it was the author's decision to ask the students to discuss these undertakings in the specially-designed chat rooms, thereby making each task more communicative.The author also decided to add further activities to each and the author discussed this idea later on in the evaluation.In order to motivate the students, the author offered bonus marks for those who participate in the discussion and extra activities.

Chapelle's scheme (2001):
For the purpose of this evaluation, it will be useful to begin with an outline of Chapelle (2001).Chapelle argues that CALL evaluation should be carried out using the theories of second language acquisition.There are two stages in her scheme: judgmental and empirical.In the judgmental stage, Chapelle (2001)  According to Chapelle (2001), however, this is not enough.She also addresses the question of what the learner actually does with the software by conducting an empirical evaluation.Whilst she focuses on different questions in each stage, she uses the same criteria in both.These criteria are: language learning potential, learner fit, meaning focus, positive impact, authenticity and practicality.The author shall judge the software by analyzing the tasks using two of Chapelle's criteria: language learning potential and learner fit.

The judgmental evaluation:
Language learning potential: Chapelle (2001) describes this criterion as the degree of 'beneficial' focus on form that the software provides to its learners.It corresponds to the following questions: does the software present students with opportunities to learn the language or just to use it?To what extent does the software shift the learners' attention towards beneficial focus on form?
Chapelle (1998) also argues that if the input has been made salient it will help with language learning.UUEG focuses intensively on the forms of the perfect tense.It promotes input saliency by highlighting these forms and writing them in italicized, bold letters.Indeed, previous research has proven such a technique to be very effective (Long and Robinson, 1998).Furthermore, both the colorful, animated pictures and the quizzes contribute to 'input enhancement' as termed by Sharwood (1993).
During the speaking task the focus is entirely on the contracted forms.In the listening and reading tasks, learners are tested on their comprehension of both the dialogue and text respectively, with a moderate focus on the forms.Chapelle (2001) and Skehan (1998) suggest some conditions which might characterize a task that draws learners' attention to the form.The author will focus on two of them-namely, 'modified interaction' and 'modified input'.
Modified interaction: Used synonymously with 'negotiation of meaning', modified interaction is when a learner's interaction is interrupted due to full or partial failure in producing or comprehending the output (Chapelle, 2001).In UUEG's exercises, however, this rarely happens, especially in the author's classes as the author decided to give each student an exercise to do alone.However, CALL advocators suggest that an activity should provide an opportunity for all learners to participate (Chapelle et al., 1996).Whilst this may be true, during the reading task the author ventured to pair the students, their performance is low though.In doing so, it would be fair to assume that negotiation of meaning might occur.First, the students were asked to read the passage (Fig. 3) and before they proceeded to the questions, the author gave them a specific period of time in which they must log into a chat room in order to discuss what they have read and identify the tenses used in the passage.Following this and in order to add more focus on form and meaning, 1 the author handed them extra questions about the passage's use of the perfect tenses.The chat conversations will be printed outmainly for task control purposes and possibly for other discussions that might be carried out later, if time allows.
Similarly, in the speaking task the students are asked to log into the chat rooms to compare their pronunciations (after they have compared their recordings with those of the model).Consequently, the author expected an interactional modification to take place.The author also devoted a portion of time to focus on irregular and regular verb forms and their pronunciation, mainly using the verbs in the program.
It is obvious that when using UUEG an interactional modification between the learners and the computer is to be expected and Chapelle (1998) suggests this to be a key element in developing a CALL task.The reading exercise (Fig. 4) provides a prime example of this theory, as meaning is expected to be Fig.5: An example of a corrected exercise broken down when students are shown the hyperlinked words.These students were expected to obtain help by clicking on each word to get its meaning.However, while this element is considered to be one of the strengths of the software, there is no other way for learners to get help with other words that they might find difficult.Therefore, in the author's opinion, a link to an online dictionary was a solution for this.
Moreover, learners were given a chance to preview the passage to help them answer the questions.By consulting the passage, learners were interacting with the computer.Interactional modification can also be achieved in the speaking task; when observing students during their performance of this exercise, it is clear that modifications can come in the form of repetition requests whilst comparing or checking the transcripts.If the software were to give a statistic of how many times options such as 'preview the passage', 'compare' and 'transcript' were accessed, it would give us a real indication of interactional modification between learners and the computer.Unfortunately, such a feature is not supported by UUEG.
Modified output: Chapelle argues that CALL software should have the ability to let students 'notice' their errors as this would help them to shift to 'a syntactic mode' that aids in internalizing the new form (1998, p. 4).Borg (1999) also claims that error awareness helps students to 'monitor and self-correct their use of language' (p.158).In UUEG, the feedback is very appropriate and one of the potential strengths of the software.By pressing the 'check answer' button that is found at the bottom of every page that has exercises, errors are crossed with a red line (or with a red cross if no answer has been given) (Fig. 5).
Chapelle (1998) also argues that learners should be given the chance to correct their errors and in the exercises discussed earlier students were given a second chance to do just this.If an error still persists, the computer will eventually display the answer in green.When the mouse is moved to the corrected answer, it flashes the error in red and the right answer in green.The author believes learners will benefit greatly from this feature.In the case of more than two errors being made, the computer will advise learners to go back to the previous charts and check their information.The author supports Chapelle (1998) view that it is advisable to have access to some online references that can help learners make corrections.
When all of the answers are correct, the software displays a 'well done' message in red at the top of the exercise and changes the answers into the color green.The colored feedback is of significance: apart from giving a focus on form, it allows the computer to take on the occupational role of teacher, as people in this profession tend to use the color red when making corrections.
A further strength of the program is the feedback provided in the test sections (Fig. 6).By pressing an orange 'e' button that appears next to each error, learners are given an explanation of each of their mistakes.However, in order to imitate the challenging conditions and characteristics of an exam, the program does not offer learners the chance to correct any errors made during the test section (unless it is uninstalled then reinstalled again).Unfortunately, there are no notifications of this in either the tests' rubrics or anywhere else in the software.
Fig. 6: A corrected test with an explanation Learner fit: In Chapelle (2001), description learner fit takes account of both the language level and its learners' characteristics.CALL materials must suit the target learners and accordingly its tasks should be set at a level that is neither too simple nor too difficult (Skehan, 1998;Chapelle, 2001).UUEG is appropriate in terms of content for learners whose levels range from lower intermediate to upper intermediate and it is designed specifically for those who want to improve their grammar in an innovative way.As for the author's students, the program is well suited to their needs.The author's claim is based on the past evaluation of the original book that has been used for more than 10 years 2 .
With regards to difficulty and control, the help section claims that there is also an 'orientation' page within the program, but the demo version used in this evaluation does not provide this facility.Nevertheless, the orientation page equips learners with the information necessary to operate the program, thus allowing them to have full control over it, which in turn gives the software more strength.Indeed, students can move freely from one section to another, record and repeat as applicable and modify their recordings whenever necessary.Furthermore, they can record as many times as they wish, as once they click the button any previous recording will be erased.
Research shows that learner control is beneficial.However, giving full control to novice learners (i.e., those with poor knowledge) might affect them in a negative way (Clark and Mayer, 2003;Hannafin andHooper 1993 in Lawler-King (2004).Whilst the majority of the exercises and their rubrics are clear and set at the correct level for the author's students, this cannot be said of those designed for error recognition.Moreover, the author has a view which is consistent with that of Heaton (1991): error recognition is not an adequate way of helping students to learn.In the author's opinion they should be exposed to the correct forms, which in turn would help them to produce the language correctly themselves.Nevertheless, this is only true when considering the first stages of learning; advanced students, the author believe, need to be able to distinguish between correct and incorrect forms.the author say this as his students still face problems with the language and still produce errors and the author doubted that these particular exercises were easy enough for them.
The tasks, like the exercises, are appropriate for teaching language at the level required.In the listening task, the dialogue is simple and the speakers talk at a suitable speed.In the reading task, the language used in the passage matches the students' abilities perfectly.The author doubted that they would encounter any difficulties in either of these two tasks as they already have been exposed to the same materials.All in all, the software presents the students with materials that are new to them and this enhances second language acquisition (Krashen, 1982;Chapelle et al., 1996).
Another issue relevant to learner fit is the level of the program's appeal to learners.If it were repetitious and dull, it might generate the unwanted factor of boredom.Yet filled with colors, different cartoon characters, animated visuals, games, drag and drop quizzes and record and compare exercises, the author considered UUEG to be very appealing and joyful.
Furthermore, the 'help' and 'report' options make this programme even more attractive.Learners can find help and support for the most frequent technical problems encountered and there is information at hand about the system requirements and how to set up the microphone (which is not easy to do).Installation instructions are also available, along with a contact number and an email address through which it is possible to leave feedback about the software.Indeed, it is the author's intention to set the author's students skill of writing and this is of great significance as the author's students are keen to improve their skills in this medium.
In the 'report' option, students can monitor their progress from one section to another within a single chapter.The report shows the learner's name alongside his or her score in each of these sections and after finishing each chapter learners can compare their most recent score with those gained earlier in the program.An overall average will then be shown at the end of the course.Characteristics and controls such as these demonstrate that UUEG makes a provision for selfstudy.
Last but not least when evaluating in terms of 'learner fit' is the cultural appropriacy of the program.It is essential that no software contradicts the values and traditions of its learners.Sheldon notes that 'publishers sometimes neglect matters of cultural appropriacy; they fail to recognize the likely restrictions operative in most teaching situations ' (1988:239).Furthermore, the Department of English Language has carefully chosen the middle-east edition of Azar (2009) because it is the only edition that does not arouse any conflict among students.As for UUEG, almost all chapters successfully matched this criterion.
What is interesting is the positive co-relational trend (Pearson = 0.900, p = 0.000) that students showed in their total marks as show in Fig. 8.That means that almost all students have gained nearly the same score in both occasions.
Students failure to succeed in this course further proves that students have serious problem with the language itself.Using technology and additional aids and materials may enhance motivation, learning aptitude and other psychological factors but still do not necessarily lead to success.The case here might be different given the nature of the sample being repeaters and administering the same treatment with other larger mixed ability sample may generate different results.

CONCLUSION
The author has analyzed a chapter from UUEG using the scheme presented by Chapelle (2001).This analysis was based on the software's suggestions and on how the author plan to use the program; thus, in Chapelle's terms, the author have carried out a 'judgmental evaluation'.
Any judgmental evaluation is based mainly on a teacher's guesswork.However, its importance lies not only in saving time and money, but also in the way teachers vary in their use of software.What one teacher does with a specific CALL task might produce poor learning results, while another teacher could use the same task positively (Jones, 1986).This is simply down to the fact that learners tend to use the program in unexpected ways that differ to the tasks' suggestions (Hosenfeld, 1976, cited in Chapelle 2001).It is for this reason that Chapelle insists on conducting an empirical evaluation.
Given the empirical analysis, UUEG seems to be a good program in the context of the author's teaching.However, the author cannot generalize this for other contexts as CALL evaluation is situation-specific (Chapelle 2001).Admittedly, UUEG has some drawbacks, yet they might be overcome in different ways-one of which is the author's previous suggestion of improving each task and by having more empirical evaluation in the same context.

Fig. 3 :
Fig. 3: The passage (for the reading task) analyses the software using two levels: the program and the teacher.In other words, she considers what learning conditions are set out by the software and what the teacher plans to do with the program respectively.

Fig. 4 :
Fig. 4: A preview of the present perfect

Fig. 7 :
Fig. 7: Students total score in the two semesters

Fig. 8 :
Fig. 8: Correlations of students' total scores in the two semesters

Table 1 :
Students scores in the two semesters

Table 2 :
Means and std.deviation for students' scores Min.