AI-generated feedback on writing: insights into efficacy and ENL student preference

Analysis

Why This Paper Matters

This paper addresses a critical and timely question in educational technology: can generative AI tools like ChatGPT effectively replace or augment human feedback in writing instruction, particularly for English as a new language (ENL) learners? With the rapid adoption of large language models in education, educators urgently need evidence-based guidance on how to leverage these tools ethically and effectively. The paper's longitudinal design and focus on both learning outcomes and learner perceptions provide a robust, balanced perspective that moves beyond anecdotal claims.

The significance is heightened by the paper's attention to ENL learners, a population that often faces unique challenges in writing development. By directly comparing AI-generated feedback to human tutor feedback in a controlled setting, the authors offer actionable insights for curriculum designers and instructors. The finding of equivalent learning outcomes is particularly important because it alleviates concerns that AI feedback might be inferior, while the near-even preference split underscores that student voice must be considered in implementation decisions.

Technical Contributions

• Longitudinal quasi-experimental design: Study 1 employed a six-week repeated measures design with pre- and post-tests, providing stronger causal evidence than cross-sectional surveys.
• Use of GPT-4 for feedback generation: The experimental group received feedback from ChatGPT (GPT-4), representing state-of-the-art LLM capability at the time of the study.
• Dual-study approach: Study 1 measured objective learning outcomes, while Study 2 captured subjective learner preferences, offering a comprehensive view of both efficacy and acceptability.
• Blended feedback framework: The paper proposes a practical model that combines AI-generated feedback (for efficiency, consistency, and immediate availability) with human tutor feedback (for nuance, empathy, and cultural sensitivity).

Results

• Study 1: No statistically significant difference in writing improvement between the AI-feedback group and human-feedback group over six weeks (p > 0.05). Both groups showed similar gains from pre-test to post-test.
• Study 2: 43 ENL learners expressed a near-even split in preference: approximately half preferred AI-generated feedback, citing its speed, detail, and non-judgmental tone; the other half preferred human feedback, valuing personalization, emotional support, and contextual understanding.
• Qualitative insights: Students who preferred AI feedback noted it was more specific and actionable, while those preferring human feedback highlighted the tutor's ability to understand their individual struggles and provide encouragement.

Significance

This paper provides crucial empirical evidence that generative AI can serve as a viable tool for writing feedback in ENL education without compromising learning outcomes. It challenges the assumption that human feedback is inherently superior and opens the door for scalable, cost-effective feedback solutions in large classes or under-resourced settings. The recommendation for a blended approach offers a practical path forward that respects both technological capabilities and human pedagogical values. For the broader AI field, this work exemplifies how rigorous educational research can inform the responsible deployment of LLMs in high-stakes learning environments, setting a precedent for future studies on AI-assisted assessment and personalized learning.