Does it really help? Exploring the impact of Al-Generated writing assistant on the students’ English writing

Regina Rahmi, Zahria Amalina, Andriansyah Andriansyah, Adrian Rodgers

Abstract


The increasing use of tools that assist English as a Foreign Language (EFL) learners in achieving writing fluency has drawn attention to the rapidly evolving role of AI in education. This study evaluates an AI-generated writing assistant in English language learning, that is the ParagraphAI text generator, focusing on its potential impact and effectiveness for L2 learners’ writing skills. This AI-powered writing software curates writing content according to writers’ preferences. Four seventh-semester EFL students were selected using homogeneous purposive sampling. Data collection involved tests and questionnaires, with subsequent analysis including text comparison to measure Lexical Diversity indices, followed by descriptive analysis. The results indicate that while the AI writing assistant aids in correcting grammatical errors and enhancing text cohesion and coherence, it lacks content density at times. In some instances, the intended message and thoughts of the students were not effectively conveyed, leading to the inclusion of ideas unrelated to the initial topic. This study underlines the importance of considering linguistic and content-related aspects in evaluating AI-generated writing assistants. While the tool enhances grammatical accuracy and structural coherence, further refinement is needed to address deficiencies in content density. The analysis of four seventh-semester EFL students offers valuable insights into the evolving AI in education, prompting considerations for optimizing these tools to better meet the diverse needs of language learners and educators.

Keywords


artificial intelligence; English as a Foreign Language; linguistic analysis; writing assistant

Full Text:

PDF

References


Abrams, L., & Davis, D. K. (2016). The tip-of-the-tongue phenomenon: Who, what, and why. In H. H. Wright (Ed.), Cognition, language and aging (pp. 13-53). John Benjamins Publishing Company.

Ary, D., Jacobs, L. C., & Sorensen, C. (2010). Introduction to research in education (8th ed.). Cengage Learning.

Bax, S. (2022). Text Inspector (June 24, 2022 version) [Online application]. https://textinspector.com/

Beers, S. F., & Nagy, W. E. (2009). Syntactic complexity as a predictor of adolescent writing quality: Which measures? Which genre? Reading and Writing, 22(2), 185-200. https://doi.org/10.1007/s11145-007-9107-5

Campbell, M. (2019). Teaching academic writing in higher education. Education Quarterly Review, 2(3), 608-614. https://doi.org/10.31014/aior.1993.02.03.92

Chen, M. X., Lee, B. N., Bansal, G., Cao, Y., Zhang, S., Lu, J., Tsay, J., Wang, Y., Dai, A. M., Chen, Z., Sohn, T., & Wu, Y. (2019). Gmail smart compose: Real-time assisted writing. In J. Tang & L. Khan (Eds.), Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (pp. 2287-2295). Association for Computing Machinery. https://doi.org/10.1145/3292500.3330723

Cheung, S. K. S. (2015). A case study on the students’ attitude and acceptance of mobile learning. In K. C. Li, T. L. Wong, S. K. S. Cheung, J. Lam, & K. K. Ng (Eds.), Technology in Education. Transforming Education Practices with Technology. Communications in Computer and Information Science (pp. 45-54). Springer. https://doi.org/10.1007/978-3-662-46158-7_5

Dizon, G., & Gayed, J., M. (2021). Examining the impact of Grammarly on the quality of mobile L2 writing. The JALT CALL Journal, 17(2), 74-92. https://doi.org/10.29140/jaltcall.v17n2.336

Durán, P., Malvern, D., Richards, B., & Chipere, N. (2004). Developmental trends in lexical diversity. Applied Linguistics, 25(2), 220-242. https://doi.org/10.1093/applin/25.2.220

Etikan, I., Musa, S. A., & Alkassim, R. S. (2016). Comparison of convenience sampling and purposive sampling. American Journal of Theoretical and Applied Statistics, 5(1), 1-4. https://doi.org/10.11648/j.ajtas.20160501.11

Gallacher, A., Thompson, A., & Howarth, M. (2018). “My robot is an idiot!”–Students’ perceptions of AI in the L2 classroom. In P. Taalas, J. Jalkanen, L. Bradley, & S. Thouësny (Eds.), Future-proof CALL: Language Learning as Exploration and Encounters – Short Papers from EUROCALL 2018 (pp. 70-76). Research-publishing.net. https://doi.org/10.14705/rpnet.2018.26.815

Gayed, M., Carlon, M. K. J., Oriola, A., M., & Cross, J. S. (2022). Exploring an AI-based writing assistant’s impact on English language learners. Computers and Education: Artificial Intelligence, 3, 100055. https://doi.org/10.1016/j.caeai.2022.100055

Hamouma, C., & Menezla, N. (2019). The impact of digital literacy proficiency on EFL students’ academic writing performance: A case study of Algerian third year EFL students. International Journal of Digital Literacy and Digital Competence (IJDLDC), 10(4), 40-55. http://doi.org/10.4018/IJDLDC.2019100103

Hanauer, D. I., Sheridan, C. L., & Englander, K. (2019). Linguistic injustice in the writing of research articles in English as a second language: Data from Taiwanese and Mexican Researchers. Written Communication, 36(1), 136-154. https://doi.org/10.1177/0741088318804821

Hwang, G. J., Xie, H., Wah, B. W., & Gasevic, D. (2020). Vision, challenges, roles and research issues of artificial intelligence in education. Computers & Education: Artificial Intelligence, 1, 100001. https://doi.org/10.1016/j.caeai.2020.100001

Ippolito, D., Yuan, A., Coenen, A., & Burnam, S. (2022). Creative writing with an AI-powered writing assistant: Perspectives from professional writers. Human-Computer Interaction. https://doi.org/10.48550/arXiv.2211.05030

Iskender, A. (2023). Holy or unholy? Interview with OpenAI’s ChatGPT. European Journal of Tourism Research, 34, 3414. https://doi.org/10.54055/ejtr.v34i.3169

Johinke, R., Cummings, R., & DiLaurao, F. (2023). Reclaiming the technology of higher education for teaching digital writing in a post-pandemic world. Journal of University Teaching & Learning Practice, 20(2), Article 01. https://doi.org/10.53761/1.20.02.01

Kellogg, R. T. (2008). Training writing skills: A cognitive developmental perspective. Journal of Writing Research, 1(1), 1-26. https://doi.org/10.17239/jowr-2008.01.01.1

Koltovskaia, S. (2020). Student engagement with Automated Written Corrective Feedback (AWCF) provided by Grammarly: A multiple case study. Assessing Writing, 44, 100450. https://doi.org/10.1016/j.asw.2020.100450

Liu, S., & Kunnan, A. J. (2016). Investigating the application of automated writing evaluation to Chinese undergraduate English major: A case study of WriteToLearn. Calico Journal, 33(1), 71-91. https://doi.org/10.1558/cj.v33i1.26380

McCarthy, P. M., & Jarvis, S. (2010). MTLD, vocd-D, and HD-D: A validation study of sophisticated approaches to lexical diversity assessment. Behavior Research Methods, 42, 381-392. https://doi.org/10.3758/BRM.42.2.381

McCrocklin, S. (2019). Learners’ feedback regarding ASR-based dictation practice for pronunciation learning. Calico Journal, 36(2), 119-137. https://doi.org/10.1558/cj.34738

McCurry, D. (2010). Can machine scoring deal with broad and open writing tests as well as human readers? Assessing Writing, 15(2), 118-129. https://doi.org/10.1016/j.asw.2010.04.002

McGee, T., & Ericsson, P. (2002). The politics of the program: MS Word as the invisible grammarian. Computers and Composition, 19(4), 453-470. https://doi.org/10.1016/S8755-4615(02)00142-1

Mohsen, M. A., & Alshahrani, A. (2019). The effectiveness of using a hybrid mode of automated writing evaluation system on EFL students’ writing. Teaching English with Technology, 19(1), 118-131.

Moore, K. A., Rutherford, C., & Crawford, K. A. (2016). Supporting postsecondary English language learners’ writing proficiency using technological tools. Journal of International Students, 6(4), 857-872. https://doi.org/10.32674/jis.v6i4.321

Nazari, N., Shabbir, M. S., & Setiawan, R. (2021). Application of artificial intelligence powered digital writing assistant in higher education: Randomized controlled trial. Heliyon, 7(5), e07014. https://doi.org/10.1016/j.heliyon.2021.e07014

Nobles, S., & Paganucci, L. (2015). Do digital writing tools deliver? Student perceptions of writing quality using digital tools and online writing environments. Computers and Composition, 38, 16-31. http://dx.doi.org/10.1016/j.compcom.2015.09.001

Patel, N. (2021, June 14). Using natural language processing for educational activities [Status update]. LinkedIn. https://www.linkedin.com/pulse/using-natural-language-processing-educational-activities-nirmal-patel/

Patterson, N. G. (2007). The devil in the machine: Problems with computerized writing assessment. Language Arts Journal of Michigan, 23(1), Article 14. https://doi.org/10.9707/2168-149X.1143

Pedro, L. F. M. G., de Oliveira Barbosa, C. M. M., & das Neves Santos, C. M. (2018). A critical review of mobile learning integration in formal educational contexts. International Journal of Educational Technology in Higher Education, 15, Article 10. http://dx.doi.org/10.1186/s41239-018-0091-4

Perry, F. (2021). The use of embedded digital tools to develop English language proficiency in higher education. Journal of Academic Language and Learning, 15(1), 1-12.

Popenici, S. A. D., & Kerr, S. (2017). Exploring the impact of artificial intelligence on teaching and learning in higher education. Research and Practice in Technology Enhanced Learning, 12, Article 22. https://doi.org/10.1186/s41039-017-0062-8

Powers, D. E., Burstein, J. C., Chodorow, M. S., Fowles, M. E., & Kukich, K. (2002). Comparing the validity of automated and human scoring of essays. Journal of Educational Computing Research, 26(4), 407-425. https://doi.org/10.2190/CX92-7WKV-N7WC-JL0A

Purcell, K., Buchanan, J., & Friedrich, L. (2013, July 16). The impact of digital tools on student writing and how writing is taught in schools [Report]. Pew Research Center. https://www.pewresearch.org/internet/2013/07/16/the-impact-of-digital-tools-on-student-writing-and-how-writing-is-taught-in-schools/

Purnama, Y. (2022). The importance of machine translation to university students. International Journal of Social Science and Human Research, 05(08), 3492-3494. https://doi.org/10.47191/ijsshr/v5-i8-19

Qian, L., Yang, Y., & Zhao, Y. (2021). Syntactic complexity revisited: Sensitivity of China’s AES-generated scores to syntactic measures, effects of discourse-mode and topic. Reading and Writing, 34, 681-704. https://doi.org/10.1007/s11145-020-10087-5

Rahimi, M., & Zhang, L. J. (2018). Writing task complexity, students’ motivational beliefs, anxiety and their writing production in English as a second language. Reading and Writing, 32(3), 761-78. https://link.springer.com/article/10.1007/s11145-018-9887-9

Scott, P., & Mouza, C. (2007). The impact of professional development on teacher learning, practice and leadership skills: A study on the integration of technology in the teaching of writing. Journal of Educational Computing Research, 37(3), 229-266. http://dx.doi.org/10.2190/EC.37.3.b

Shermis, M. D., Burstein, J., Higgins, D., & Zechner, K. (2010). Automated essay scoring: Writing assessment and instruction. In P. Peterson, E. Baker, & B. McGaw (Eds.), International encyclopedia of education (3rd ed., pp. 20-26). http://dx.doi.org/10.1016/B978-0-08-044894-7.00233-5

Silva, T. (1993). Toward an understanding of the distinct nature of L2 writing: The ESL research and its implications. TESOL Quarterly, 27(4), 657-677. https://doi.org/10.2307/3587400

Sumakul, D. T. Y. G., Hamied, F. A., & Sukyadi, D. (2021). Students’ perceptions of the use of AI in a writing class. Advances in Social Science, Education and Humanities Research, 624, 52-57. https://doi.org/10.2991/assehr.k.220201.009

Terblanche, N., Molyn, J., Williams, K., & Maritz, J. (2022). Performance matters: Students’ perceptions of artificial intelligence coach adoption factors. Coaching: An International Journal of Theory, Research and Practice, 16(1), 100-114. http://dx.doi.org/10.1080/17521882.2022.2094278

Tsiriotakis, I. K., Vassilaki, E., Spantidakis, I., & Stavrou, N. A. M. (2017). The examination of the effect of writing strategy-based procedural facilitative environments on students’ English language writing anxiety levels. Frontiers in Psychology, 7, 2074. https://doi.org/10.3389/fpsyg.2016.02074

Wang, F., & Wang, S. (2012). A comparative study on the influence of automated evaluation system and teacher grading on students’ English writing. Procedia Engineering, 29, 993-997. https://doi.org/10.1016/j.proeng.2012.01.077

Wingate, U. (2010). The impact of formative feedback on the development of academic writing. Assessment & Evaluation in Higher Education, 35(5), 519-533. http://dx.doi.org/10.1080/02602930903512909

Wolfersberger, M. (2003). L1 to L2 writing process and strategy transfer: A look at lower proficiency writers. TESL-EJ, 7(2).

Woo, M., Chu, S., Ho, A., & Xuanxi, L. (2011). Using a wiki to scaffold primary-school students’ collaborative writing. Journal of Education Technology & Society, 14, 43-54.

Zawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019). Systematic review of research on artificial intelligence applications in higher education–Where are the educators? International Journal of Educational Technology in Higher Education, 16, Article 39. http://dx.doi.org/10.1186/s41239-019-0171-0

Zhao, X. (2022). Leveraging artificial intelligence (AI) technology for English writing: Introducing Wordtune as a digital writing assistant for EFL writers. RELC Journal, 54(3), 890-894. https://doi.org/10.1177/00336882221094089




DOI: https://doi.org/10.24815/siele.v11i2.35875

Article Metrics

Abstract view : 0 times
PDF - 0 times

Refbacks

  • There are currently no refbacks.


Print ISSN: 2355-2794, Online ISSN: 2461-0275

This work is licensed under a Creative Commons Attribution 4.0 International License.


View Journal Stats