Do tutors matter ? Assessing the impact of tutors on first-year academic performance at a South African university

* Department of Environmental Science, University of South Africa. Email: mckaytjm@unisa.ac.za Abstract This research sought to determine if a teaching intervention using tutors in a South African university could promote epistemological access to university for first-year students. Although hiring, developing and managing tutors takes money, time and energy, the effectiveness of tutors in the South African context is underreported. The first-year class under study was diverse in terms of gender, race, ethnicity and geographical origin. The tutors were all postgraduate students, and similarly diverse. In terms of research design, student test results were compared from one test to another. The students also rated the tutors. Students who attended the majority of the assigned tutorials improved their marks by an average of 20%. Even students whose tutorial attendance was haphazard fared better academically than those who did not attend at all. Students who skipped all the tutorials saw a dramatic decline in their marks, suggesting that tutorial attendance should be obligatory. Individual tutors matter, however. It seems that some tutors can explain, facilitate understanding and engage their students better than others. Students assigned to such tutors achieved the greatest academic gains. Thus, recruitment strategies and tutor training are crucial. Tutor popularity (based on student ratings) did not correlate with positive academic improvements. Thus, student ratings should not by themselves strongly influence hiring decisions. In conclusion, resources allocated to tutors were worthwhile and the tutors enabled epistemological access for many.


Introduction
One of the characteristics of higher education in South Africa is the poor pass rate of firstyear students, with between one in three and two in three students failing at least some modules (Badat, 2010;Coughlan, 2006;Du Plessis, Müller & Prinsloo, 2005;Eiselen & Geyser, 2003;Grussendorff, Liebenberg & Houston, 2004;Lourens & Smit, 2003). Multiple reasons for this failure have been posited. These include: a mismatch between home language and the language of learning and teaching (LOLT); students lacking the sociocultural skills and knowledge (such as academic literacy) required for success in higher education; poor time management; poor orientation strategies; and poor study habits (Cross & Carpentier, 2009;Grant, 2005;Jansen, Sehlapelo & Tabane, 2007;Maree, 2008;Mgqwashu, 2009;Mouton, Louw & Strydom, 2013). Such students are often described as "underprepared for university study" (Davids, 2014;Dube, Kane & Lear, 2012;Underhill & McDonald, 2010). Ensuring that students gain epistemological access to the knowledge domains of academia requires an adaptation to the teaching programme to help students acquire the requisite skills, content and ways of doing specific to the discipline (Underhill, Clarence-Fincham & Petersen, 2014). This includes overt teaching of the conventions of academia, where students are "socialised" into the discipline of study (Archer, 2008). One way of doing this is to make use of structured tutorials run by discipline-specific tutors.
As tutors and tutorial groups can be viewed as a means of bringing the key elements of participation and "reification" together, the tutor intervention study described here drew on the community of practice approach (Davids, 2014;Grant, 2005;Lave & Wenger, 1991). "Reification" is understood here as the "way things are done" and is manifest in specific academic activities; participation is the performance of the way "things are done" (Wenger, 1998). In this case, the tutorials and the tutors created a learning environment in which students could learn the way things are done through performance in a safe, supportive environment (Sutherland, 2009;Aluko & Hendrikz, 2012). The tutorial activities can be thought of as boundary objects -structured learning activities designed to enable students to move progressively towards full participation in a community of practice (Wenger, 1998). However, the structured learning activities are insufficient on their own: they require brokering by the tutors. Page, Loots and Du Toit (2005) and Underhill, Clarence-Fincham and Petersen (2014) both argue strongly that these tutors must be part of the discipline themselves if this "brokering" is to be successful.

Tutorials and tutors
There are various types of tutoring: (1) one-on-one; (2) peer; (3) group; (4) cross-age; and (5) online, using email or discussion forums (Van Lehn, 2011). In this study, the type of tutorial system could be best described as a small group or class in which a more experienced or able student works with less experienced students to help them, in a structured and organised manner (such as with set meeting times and venues), to adjust to a new environment or discipline (Page, Loots & Du Toit, 2005). Such tutorials enable a more individualised instruction, considered vital in the context of large classes, complex student needs and a diverse student body (Underhill, Clarence-Fincham & Petersen, 2014). They were also broadly similar to Geography tutorials described by Appleton (2010) as teaching sessions aimed at helping students understand complex power relations and writing skills, by making use a variety of sources such as books, websites and reports. Tutorials are, therefore, small groups or classes in which learning is more personalised and individualised. They complement the lectures (Mischo & Haag, 2002).

The international perspective
A great deal of literature supports the notion that tutoring can have a positive effect on student academic performance (Beck, Skinner & Schwabrow, 2013;Cohen, Kulik & Kulik, 1982;Colvin, 2007;Merrill et al., 1992;Robinson, Schofield & Steers-Wentzell, 2005). For example, Mischo and Haag's (2002) work in Germany found that tutoring positively affected school performance and motivational levels. Margolis (2005) found that exposure to tutoring resulted in better test scores, enabled students to acquire more learning strategies, and improved their reading skills. Walsh, Larsen and Parry (2009) found that both academic tutors (who helped students with the discipline) and personal tutors (who helped them to adjust to university life) had a positive effect on first-year student success. In a cross-national study, Hof (2014) found that tutoring had a positive impact on academic performance in India, Israel, Italy and the United States. With tutoring, student's tests scores go up, their reading improves, and their ability to understand mathematics, language and social science concepts increases (Hof, 2014;Robinson, Schofield & Steers-Wentzell, 2005). Thus, employing tutors in higher education offers a significant value proposition (Retna, Chong & Cavana, 2009;Truer, 2014). This is especially true for universities battling with large classes and at-risk students (Colvin, 2007;Retna, Chong & Cavana, 2009;Robinson, Schofield & Steers-Wentzell, 2005).
While the literature calls for the need for well-planned tutorials, there is also strong support for the proper training and careful appointment of tutors (Sutherland, 2009). For example, Margolis (2005) emphasises that tutors who are certified (qualified), experienced and competent are more successful than ones who are not. Thus, expertise and skills matter. In particular, good tutors help students to build self-efficacy, encourage persistence and take credit for achievements. Walsh, Larsen and Parry (2009) suggested the best tutors make themselves available to their students and are friendly, approachable and sympathetic. Van Lehn (2011) argues that good tutors are ones who are able to scaffold reasoning tasks (using guided prompting) and give frequent and motivational feedback (praise the efforts of the students; get students to focus on effort, not ability). Scaffolding also requires that a tutor is able to unpack the task/reading; break tasks down into smaller manageable components; make tasks shorter, less complex, and more concrete; and issue clear instructions (Appleton, 2010).
Good tutors, therefore, give students repeated opportunities to master a skill. They are strategic in their focus -that is, on the important skills/content. They use simple, direct language and avoid negative comments. They have good social skills as they are enthusiastic, good listeners, have good/positive body language and actively manage anxiety in their students (Appleton, 2010). Tutors are unlikely to be simply born with such skills and pedagogical knowledge, so tutors need structured support and training -workshops, survival guides, definite roles and responsibilities, tutor support groups and mentoring (Sutherland, 2009). Robinson, Schofield and Steers-Wentzell (2005) found that tutor training had a positive effect by making the tutors more interactive; training improved their interpersonal, managerial and content skills (especially teaching tutors to stay on task and being well versed in the content). Nevertheless, tutors also need to be chosen well, monitored, and adequately supervised (Appleton, 2010;Margolis, 2005). Despite this, tutors are often part-time, motivated only by the money, inexperienced, and usually students themselves. Tutors, then, are beginners, lacking in subject knowledge and teaching experience, yet under pressure to produce high-quality learning experiences.

Tutors: the South African perspective
A number of studies published between 2005 and 2014 indicate that many South African universities are grappling with tutor-related issues. A number of themes emerge from these studies. First, within the context of massification, a tutorial system is seen as pivotal to broadening participation and improving throughput rates of underprepared students (Fouche, 2007;Page, Loots & Du Toit, 2005;Thomen & Barnes, 2005;Underhill, Clarence-Fincham & Petersen, 2014). Second, tutorials seem to have a positive impact on academic performance -so much so that skipping tutorials was linked to academic failure (Fouche, 2007;Page, Loots & Du Toit, 2005). Fouche (2007) and Seabi, Cockcroft and Fridjhon (2009) found a statistically significant difference between the academic performance of students who attended tutorials and those who did not. Third, much work has been conducted on the value that tutorials offer students, such as promoting deep learning, enabling active engagement, lowering student stress levels, building cognitive strengths that support the acquisition and retention of skills and knowledge, and promoting a passion for the discipline (Hlatshwayo, 2013;Thomen & Barnes, 2005;Underhill, Clarence-Fincham & Petersen, 2014). Fourth, a strong emphasis on the need to develop and train tutors, specifically within the discipline in which they must operate, emerges (Dube, Kane & Lear, 2012;Roux, 2009;Underhill, Clarence-Fincham & Petersen, 2014;Underhill & McDonald, 2010).

The research context
The students who participated in this study were all first-year Geography students at the University of Johannesburg. The class was diverse in terms of race, gender, socio-economic status, and degrees for which students were registered. A typical student in this module could be described as black, male, aged roughly 19½, having matriculated in a rural or township school in Gauteng, where the fees were nil or low but the matriculation pass rate was relatively good. The tutorials were designed explicitly to support the lectures but not to replace or repeat them. Tutors were to assist students with course-related Geography knowledge but -in line with the recommendations of Roux (2009) -without re-teaching the lectures and without doing all the work for the student. Tutorials supported the lectures by providing the students with additional information, through case studies or examples, of the content with the purpose of promoting greater understanding thereof (see Appendix I for an example from the lecture/work schedule).
Also, the tutorials were designed to support the acquisition of specific academic literacy skills, such as "reading for understanding", academic writing, problem-solving, and how to build an academic argument. Thus, the tutorials were highly structured, with specific reading and writing tasks that students had to complete. This type of tutorial was developed based on earlier work done by the author (McKay, 2013). Students signed up for specific time slots, which were then allocated to specific tutors. In line with the recommendations of Roux (2009), all tutors were supposed to have the same number of students and students were not supposed to switch tutor groups. To "incentivise" the students to attend and do the learning activities, students were told that the tutorials were compulsory and that work completed for the tutorials would count 10% towards the semester mark. There were 13 tutorials over the 14-week semester, each with an associated task. Each student's 10 best tutorial tasks were incorporated into the final semester mark (similar to what was reported by Thomen and Barnes [2005], where tutorial marks counted 5%). Unfortunately, as with the case of Fouche (2007), and unlike the situation reported at the University of Fort Hare by Thomen and Barnes (2005), there was no way to force students to attend tutorials.
The tutors were all postgraduate Geography students of mixed gender and race. In line with the recommendations of Roux (2009), there was a tutor co-ordinator who was a doctoral student in Geography. The coordinator assisted the lecturer with academic administration -capturing marks, checking that tutorials took place, assisting with the management of the tutors, giving feedback to the lecturer when problems arose, and so forth. The tutors had a number of duties: (1) attending tutor meetings and training sessions; (2) sitting in on lectures (to assist the lecturer to maintain order in the lecture hall); (3) handing out and collecting documents (such as tests and assignments); (4) preparing for the tutorial sessions; (5) being punctual for tutorials, using the full teaching time; (6) various administrative tasks (such as assisting with capturing marks and keeping attendance records); (7) the marking of some tasks; (8) scheduling individual one-on-one sessions with students (on request); and (9) referring students with general study habit problems, writing problems, emotional and other problems to the relevant professionals (such as the Writing Centre or Student Psychological services).
Although tutorial support for this module began in earnest in 2010, to date there are no official time slots for tutorials (due to timetable and venue constraints). The tutorial model was refined over time. In 2011, not all the tutors taught all the scheduled tutorials or submitted reports. This could be because once a tutor was on the payroll, he or she was paid regardless of whether he or she had delivered the service. Thus, some tutors realised that they could skip tutorials without losing out financially. In addition, if tutors resigned, they could not be replaced as the budget had already been allocated to a specific individual and reallocation of these funds was an overly bureaucratic task that often yielded no result. Another challenge was that not all students attended the tutorials, and tutors reported that even when students attended, they were seldom prepared for them. Moreover, informal complaints from students indicated that some tutors seemed to have better teaching and facilitation skills than others. Consequently, changes were implemented in 2012. Tutor payments were switched to the end of each term and only upon the submission of a claim form. Tutors had to sign contracts outlining their duties and what was expected of them. Tutors could no longer give 24 hours' notice -they could only leave at the end of a term or semester. Contracts were only issued for one semester at a time to manage poorly performing tutors out of the system.
For years, poor salaries hindered the recruitment of quality tutors. Thus, over time, payment rates were improved, with the rate per hour moving from R65 in 2011 to R109 by 2013. The number of hours for which the tutors worked also increased from five to ten per week, to increase the "take-home pay" of the average tutor. A cause of disagreement for the tutors was that under the old system hourly rates were based on qualifications, not job description. For example, in 2011, a tutor with an undergraduate degree was paid R65 an hour, one with an honours degree R75 an hour, and one with a master's degree R88 per hour. Tutors considered this system unfair and wanted equal pay for equal work. In 2014, changes were made and all tutors were paid the same rate of R109 per hour for 140 hours (or roughly R15 200 per semester). This is in line with Page, Loots and Du Toit (2005), who recommend paying tutors the same basic fee (although with a bonus system for those who "went the extra mile"). The tutor coordinator received double that, as he or she worked 20 hours a week. Overall, the total tutor budget for the first semester in 2014 was about R90 000 (an increase of 53% from 2011). In addition, stricter hiring criteria were adopted to ensure that only tutors who had an affinity for teaching were hired.

Research method
To determine whether the tutors had a positive effect on students' academic performance, the following methodology was employed. First, a class list was drawn up to constitute the population. As the official class list can change from day to day and week to week, the lecturer had to develop a class list using class attendance forms, test scripts, assignment submissions and tutorial attendance registers. On this basis, some 358 students constituted the total population. However, only 336 were on the final official faculty-issued class list. When the class list and the final official list matched one another by the end of the semester, a discrepancy of 6% (22 students) was recorded. It may be that some students attend classes and write tests but do not register or cancel their registration. The study followed the methodology used by Fouche (2007), Mischo and Haag (2002), and Seabi, Cockcroft and Fridjhon (2009) -one in which the first test was used as a pre-test (or baseline test) and the second test constituted the post-test. Only students who wrote both tests were included in the sample (323 students). Those students (numbering 243) who had consistently attended tutorials (at least 10 of the 13 tutorials) were placed into Group One. The second grouping (Group Two) comprised students who attended only a few tutorials (64 students); and the third group (Group Three) represents students who had attended no tutorials (11 students). Results of the tests for each of these three groups were compared to establish if there had been an improvement.
In order to enable students to rate their tutor, a rating sheet was developed using a Likert scale (see Appendix II). In terms of approval ratings, 238 students participated in the opinion survey. The opinion survey was conducted during the last week of the semester during one of the lecture periods, and yielded an overall response rate of 71% of those on the final official class list (a few students who did not attend any tutorials also rated the tutors, but these scores were not included in the analysis). No student who was not on the official class list submitted an opinion survey. The survey was not anonymous, as the survey instrument asked for their student number -but they could leave it off it they wanted to. Students were assured that they could be entirely honest in their evaluation of the tutor as there would be no ramifications either for the tutor or for themselves. The scores were added up and converted to a percentage. This methodology meant that no tutor could score less than 20%, but could score up to 100%. The rating sheets were then captured in a way that generated an average rating for each tutor.

Research findings and discussion
Comparing the results between Test One and Test Two for each three groups (Table 1) makes it clear that the tutorials helped the students. Even attending some tutorials was better than attending no tutorials. The marks of students who attended 10 or more tutorials (Group One) marks went up, on average, by 20%. The marks of students who attended some tutorials (Group Two) fell by an average of 8%. Students who attended no tutorials (Group Three) saw their marks decline by an average of 13%. As the class average for Test One and Test Two was 46% and 45% respectively, it can be deduced that students who did not attend tutorials, or who attended only a few, placed significant downward pressure on the class average.

Table 1: Test result comparison between the three groupings Tutorial groupings by attendance
Change between Test 1 and Test 2 Group One (attended 10 or more tutorials) Mean increase +20% Group Two (irregular tutorial attendance) Mean decrease -8% Group Three (did not attend tutorials) Mean decrease -13% An interesting, yet unexpected, finding was that there was a large difference between some of the tutors when a deeper analysis was undertaken of Group One students. For example, as shown in Table 2, two tutors (Tutor A and Tutor B) stood out as having the most positive effect on the marks of the students in their tutorial group. Students tutored by tutors A and B recorded an average increase between Test One and Test Two of 29%. One tutor (Tutor E) was found not to have made a positive difference; students tutored by this person recorded a decline in their test marks by 2% on average. Nevertheless, when compared to Group Two and Group Three, it was found that it is better to have a weak tutor than no tutor at all. Of interest, students also voted for one of the good tutors with their feet. That is, they had left the weakest tutor's class and joined the class of the strongest tutor. Accordingly, the strongest tutor was overwhelmed with student numbers (77), whereas the weakest tutor was left with only 42 students. Tutor A and Tutor B also had the best attendance records -indicating that they seemed to be more effective than other tutors in getting students to attend tutorials. Analysis of the results of the opinion survey (Table 3) reveals that student approval ratings do not correlate with improved academic performance [r 2 = 0.0022]. This finding may explain why Davids (2014) could not determine whether a tutorial system is worthwhile because that study used an opinion or rating survey. Table 3 shows that the approval rates were all uniformly high (the average rating was 83%) and clustered. One tutor (Tutor B) was given an extremely low rating by 17% of the class, despite the fact that this tutor had a positive impact on test scores. The tutor deemed the worst (Tutor E) had the second highest approval rating. That said, Tutor A, deemed to be one of the best tutors by the test scores, was also rated the best by the students. Tutor A also had the largest number of students responding -again, reinforcing the possibility that a good tutor inspires students to be more engaged and attend classes. Overall, with only five tutors to evaluate, it is not possible to say that race, gender and level of qualification of the tutor had an effect on the difference in test scores or the results of the opinion survey. Attending tutorials had a positive effect on the academic performance of the students. It must be stressed, however, that these tutorials were highly structured. Each one was carefully constructed as a unique learning opportunity to support the acquisition of conceptual and content understanding, as well as academic literacy. Thus, it cannot be assumed that tutorials not similarly structured will have the same positive effect. The actual tutor also matters. Some tutors are better able to explain, facilitate understanding, and encourage their students to become engaged learners. These tutors saw their students make the biggest gains between Test One and Test Two. The findings highlight that a poor or weak tutor can negatively affect learning. When considering both tutorial attendance and participation in the opinion survey, the results indicate that a good tutor may also stimulate and encourage participation in the module. In a situation where the class is big, students are new to the university and the lecturer is overloaded work-wise, the support of tutors and the tutor coordinator proved valuable and the expenditure justifiable.

Concluding remarks
Overall, this study supports the work of Fouche (2007), Page, Loots and Du Toit (2005), Thomen and Barnes (2005), and Underhill, Clarence-Fincham and Petersen (2014). In the case of this first-year class, tutors played an important role in promoting student academic success. As the tutorials were carefully structured, however, this may also have contributed to the academic success of the students. In line with Fouche (2007) and Page, Loots and Du Toit (2005), skipping tutorials was linked to academic failure. Thus, it is recommended that tutorials be made compulsory. This study also indicates that, as noted by Davids (2014), only people with the appropriate mix of qualifications, facilitation skills and popularity be employed as tutors. Certainly, the most popular tutor is not necessarily the best one. There is a need to develop and train tutors, because there was a discernible difference in skill and competence. Training may not be sufficient; tight management of who is appointed and how they conduct themselves is required to ensure that poor tutoring does not disadvantage students. Thus, accountability and reward should be built into the tutor system. Good tutors need to be paid well and retained. Perhaps programmes to retain good tutors for longer than a semester or an academic year should be explored.

Appendix I: Example from the lecture/work schedule
Week 1 Tut 1 Gauteng as a gateway to Africa The purpose of this tutorial is to help you understand the importance of: (a) using a number of different sources to inform yourself; (b) using sources from a variety of places or writers; and (c) knowing the difference between information written by corporations or employees for the purpose of selling something (a product or an idea) and information written by academics.

Contextual analysis
Week 2 Tut 2 Exploring social grants in South Africa The purpose of this tutorial is to explore the pros (advantages) and cons (disadvantages) of social grants. While it is obvious that in the face of poverty, social grants play a vital role in helping people improve their quality of life in South Africa, there are significant downsides to having a population dependent upon welfare.

Read text and figures and graphs and answer questions
Week 6 Tut 6 What are the environmental consequences of the massive rise of manufacturing in China?
Here, we trace the environmental and human health consequences of China's massive economic growth. This growth has been based primarily on manufacturing, with China effectively becoming the "workshop of the world". However, this economic success story has come at huge environmental and human health costs. Some argue that if China were a true democracy, then the government would have to pay far more attention to solving this problem than they currently do.
Essay writing