Talk title: Evaluating writing in the age of AI: what’s left to assess?
Abstract: The fields of language education and language assessment have long incorporated and accounted for novel technologies to improve the practices of teaching and testing. With the appearance of the writing technologies from the typewriter up to grammar checkers like Grammarly, language testers found ways to refocus writing assessment on higher-order cognitive skills and more involved genres. However, with openly available AI tools such as ChatGPT, even rhetorical structure, cohesion, and unity can all be automated within conventional templated writing formats. At this time, the field of language testing is behind in understanding how this tool hurts or helps assessment of the writing construct. The current study seeks to understand how AI chat tools produced prompted writing, how automatically generated essays can be differentiated from human-generated essays, and given this differentiation, what qualities are important to assess in writing moving forward. Fifty essays generated by ChatGPT were compared to 250 human-written evidence-based essays using moves analysis, as well as fine-grained automated measurement of linguistic features. These features are used to qualitatively and quantitatively distinguish essays of both types. Finally, higher-scoring human-generated essays are investigated more closely to understand what is valued in human writing. This will set the stage for further investigating human rater perceptions of text quality and text providence.
