Using technology-assisted peer feedback to improve academic writing

The poster discusses the possibilities of technology-assisted peer feedback in English as a Foreign Language (EFL) writing classrooms among low proficiency students. It is a part of an ongoing research project developed after a study conducted in the first half of 2019 (Lazic & Tsuji, 2020a, 2020b). The first goal is to explore the effectiveness of in-class activities, which include technology-assisted peer feedback, in improving global aspects of writing, i.e. paragraph structure and content, and to examine the uptake of peer feedback delivered via an Automated Writing Evaluation tool (AWE), Educational Testing Service (ETS) Criterion®. Second, the study looks at students’ perceptions. Participants were 15 first-year students taking an academic writing class.


Introduction
Potentials and drawbacks of peer feedback are well documented (Allen & Katayama, 2016;Hyland & Hyland, 2006). In the Japanese context, studies explored the use of AWE in writing classes (Koizumi, Asano, & Agawa, 2016;Wakabayashi, 2013). In general, less attention was paid to combined AWE and peer feedback's potential in improving writing (Stevenson, 2016).
To address these issues, we first conducted a study in 2019 to explore students' perceptions about combined AWE and peer feedback (Lazic & Tsuji, 2020a) and look at the effects of the engagement with this type of feedback on revision uptake (Lazic & Tsuji, 2020b). Based on the recommendations of these two studies and the literature review, the current study asked the following Research Questions (RQ).

RQ1
. What is the effectiveness of in-class peer feedback activities delivered via AWE in improving global aspects of writing?
RQ2. What are the students' perceptions of in-class peer feedback activities delivered via AWE?

Method
Fifteen Japanese EFL learners studying at a Japanese public university participated in the study after signing a consent form approved by the university's ethics committee. This writing course aims to teach writing academic paragraphs, coherence, and content development. Classes met once a week, and each lesson lasted for 90 minutes.
The study used ETS Criterion® (Burstein, Tetreault, & Madnani, 2013). After the initial training on the use of this AWE, students were introduced to one activity per class with the purpose of practicing giving feedback while focusing on the paragraph's organization and content (15 minutes). Then, students used ETS Criterion® to provide peer feedback on paragraphs written at home (30 minutes). Finally, students had to make revisions based on peer and ETS Criterion®'s comments (30 minutes). The class ended with writing a short reflection and answering students' questions (15 minutes). The procedure was repeated during Weeks 4, 5, and 6 introducing a different activity each week: a textbook peer feedback form, group review of one student writing sample (a form designed by the instructor), and the use of polite and other useful expressions when giving feedback. The instructor gave feedback at the end of the course.
Students' reflections were collected after each session and translated into English. To discover trends in students' reflections, we used KH Coder, text-mining software (Higuchi, 2016). To detect emerging topics, a co-occurrence network of words was used. Writing samples and students' comments (on three different occasions) were downloaded from ETS Criterion®. When analyzing the data, descriptive statistics were used (note: due to using Excel, some of the percentages are rounded to the nearest percent and might not add up).
These scores show improvement or lack of it. Writing improved during Writing 2 and 3, although moderately. In the table, this is a difference between the score frequencies across different score categories (from Advisory to 6). For example, in Writing 2, for their 'before' writing, most students got a score of '1'. When students rewrote based on peer and ETS Criterion® feedback, the number of scores '1' decreased, while the number of students who got score '2' increased. During Writing 2, students wrote a paragraph about whether single-sex education is obsolete or not, a topic chosen from the ETS Criterion® topic bank. When giving feedback on this writing, students commented more on the global aspects of writing than on any other issue. In other studies (e.g. Allen & Katayama, 2016), comments on grammatical errors were the bulk of peer feedback.
Writing samples were qualitatively analyzed to look at the actual changes and uptake of comments delivered by peers via an AWE tool. Like the results of score analysis (Table 1), this analysis shows that revisions were minimal, and at surfacelevel, e.g. spelling. In terms of adding new ideas and improving the structure and the coherence of the paragraphs, students made few changes overall. Due to a lack of space and as most students generally made minimal changes, only one writing sample is used as representative. Figure 1 shows where the student addressed one surface-level mistake identified by her peer (the mistake and change are underlined). Out of 11 comments received, some referred to the content. For example, one classmate expressed her surprise about a piece of information (Figure 2, a rectangle) but did not suggest any changes. This writing was graded as unsuccessful as the student did not support the main idea in the paragraph. The low revision uptake and minimal text changes are possibly due to limited revision time, insufficient and inadequate peer comments, and low proficiencies (TOEFL ITP 2 scores between 400 and 450). Others found that lower-level proficiency and less confident learners comment and revise less (Allen & Katayama, 2016;Wakabayashi, 2013). Co-occurrence network analysis of the 46 most frequent words was used to answer RQ2 ( Figure 2). The size of a circle indicates the word frequency. Circles of different colors are grouped into communities based on their modularity degree -a measure used to detect themes in the data (Higuchi, 2016). Pronouns were included to understand what students focused on: frequent use of 'I/my' indicates that students were more interested in using the activity to improve their writing. Identified themes are: students reflecting on problems in their writing and how the activity made writing better (purple circles); reading helped students to think about writing (red); feedback was helpful with rewriting grammar and getting a higher score (green); and ability to notice (problems) as their peers pointed them out (yellow).

Conclusions
Learners engaged in the AWE supported peer feedback activities by providing more comments on content and paragraph structure, which is different from the previous studies. Besides, as seen from the reflection analysis, students found the activity useful, e.g. by reading their peers' writing, they understood their writing problems better. However, feedback uptake was minimal, as presented in the changes in ETS Criterion® scores and qualitative analysis. Although changes were minor, compared to the previous study (Lazic & Tsuji, 2020b), students did focus on global aspects of writing when commenting. Thus, it may be concluded that the combined activity can be used for the student benefit, but with modifications, e.g. a longer time for revisions. Consequently, the next research question to answer is: why did learners not make substantial changes to their texts?