Teachers’ made writing assessment instrument: Pedagogical insight from process to product

Khusnul Khotimah, Deisyi Anna Batunan, Nur Mukminatien

Abstract


Despite the urgency of assessment instruments in teaching foreign language writing, evidence regarding how teachers develop assessment instruments and maintain their quality, particularly for comparison and contrast essay writing based on a genre-based approach to teachers’ classroom practices, is scantily reported. To address this empirical need, this article showcases how a set of teacher-made assessment instruments for a comparison and contrast essay based on a genre-based approach in a writing essay course at a public university in East Java was carefully and deliberately developed. To attain the research objective, this research enacted design-based research (DBR) inspired by Cavallaro and Sembiante (2021). In addition to content validity, face validity and inter-rater reliability were ensured by involving ten students, three fellow writing teachers, and two trained raters. This assessment instrument development project produces a set of valid formative assessment instruments, comprising a writing prompt and scoring rubric. The inter-rater reliability shows a high degree of positive correlation with a reliability coefficient of 0.862 from a correlational analysis based on Pearson correlation coefficient. This in-practice assessment instrument development can be used as a reference, contributing to the teachers’ assessment literacy development. It also serves as a valuable model for future researchers to develop and expand nested classroom assessment scholarship. A well-structured validity and reliability process of assessment instrument development will practically enhance the teaching practices in the essay writing course in particular and EFL teaching in general.

Keywords


classroom assessment; comparison and contrast essay; teacher-made assessment instrument; teachers’ assessment literacy; writing assessment

Full Text:

PDF

References


Alderson, J. C., Brunfaut, T., & Harding, L. (2017). Bridging assessment and learning: A view from second and foreign language assessment. Assessment in Education: Principles, Policy & Practice, 24(3), 379-387. https://doi.org/10.1080/0969594x.2017.1331201

Badger, R., & White, G. (2000). A process genre approach to teaching writing. ELT Journal, 54(2), 153-160. http://doi.org/10.1093/elt/54.2.153

Bloxham, S., den-Outer, B., Hudson, J., & Price, M. (2016). Let’s stop the pretence of consistent marking: Exploring the multiple limitations of assessment criteria. Assessment & Evaluation in Higher Education, 41(3), 466-481. http://dx.doi.org/10.1080/02602938.2015.1024607

Blumberg, B., Cooper, D. R., & Schindler, P. S. (2005). Business research methods. McGraw-Hill Education.

Brown, H. D. (2004). Language assessment: Principles and classroom practices. Pearson Education.

Brown, H. D., & Abeywickrama, P. (2019). Language assessment: Principles and classroom practice (3rd ed.). Pearson Education Inc.

Brown, J. D. (1995). The element of a language curriculum. Heinle & Heinle.

Callaghan, M., & Knapp, P. (1989). The discussion genre. Metropolitan East Disadvantaged School Program.

Caplan, N. A., & Farling, M. (2017). A dozen heads are better than one: Collaborative writing in genre-based pedagogy. TESOL Journal, 8(3), 564-581. https://doi.org/10.1002/tesj.287

Cavallaro, C. J., & Sembiante, S. F. (2021). Facilitating culturally sustaining, functional literacy practices in a middle school ESOL reading program: A design-based research study. Language and Education, 35(2), 160-179. https://doi.org/10.1080/09500782.2020.1775244

Creswell, J. W. (2005). Educational research: Planning, conducting, and evaluating quantitative and qualitative research. Pearson Merrill Prentice Hall.

Davies, A. (2008). Textbook trends in teaching language testing. Language Testing, 25(3), 327-347. https://doi.org/10.1177/0265532208090156

DeLuca, C., LaPointe-McEwan, D., & Luhanga, U. (2016). Approaches to classroom assessment inventory: A new instrument to support teacher assessment literacy. Educational Assessment, 21(4), 248-266. https://doi.org/10.1080/10627197.2016.1236677

DiDonato-Barnes, N., Fives, H., & Krause, E. S. (2013). Using a table of specifications to improve teacher-constructed traditional tests: An experimental design. Assessment in Education: Principles, Policy & Practice, 21(1), 90-108. https://doi.org/10.1080/0969594X.2013.808173

Effendi, T., & Mayuni, I. (2022). Examining teacher-made English test in a language school. LADU: Journal of Languages and Education, 2(2), 67-76. https://doi.org/10.56724/ladu.v2i2.109

Englert, C. S., Raphael, T. E., Anderson, L. M., Anthony, H. M., & Stevens, D. D. (1991). Making strategies and self-talk visible: Writing instruction in regular and special education classrooms. American Educational Research Journal, 28(2), 337-372. https://doi.org/10.3102/00028312028002337

Field, A. P. (2005). Discovering statistics using SPSS. Sage Publications Inc.

Firkins, A., Forey, G., & Sengupta, S. (2007). Teaching writing to low proficiency EFL students. ELT Journal, 61(4), 341-352. http://dx.doi.org/10.1093/elt/ccm052

Fitriyah, I., Basthomi, Y., Khotimah, K., & Gozali, I. (2022). Implementation of assessment for learning in online EFL writing class: A case of novice undergraduate teachers. LEARN Journal: Language Education and Acquisition Research Network, 15(2), 129-159.

Fitriyah, I., Gozali, I., Widiati, U., El Khoiri, N., & Singh, A. K. J. (2023). EFL writing teachers’ practices and values of assessment for and as learning in a constrained context. Computer-Assisted Language Learning Electronic Journal, 24(2), 104-128.

Ghanbari, N., & Barati, H. (2020). Development and validation of a rating scale for Iranian EFL academic writing assessment: A mixed-methods study. Language Testing in Asia, 10, Article 17. https://doi.org/10.1186/s40468-020-00112-3

Glušac, T., & Milić, M. (2022). Quality of written instructions in teacher-made tests of English as a foreign language. English Teaching & Learning, 46(1), 39-57. https://doi.org/10.1007/S42321-021-00079-1

Hammond, J., Burns, A., Joyce, H., Brosnan, D., & Gerot, L. (1992). English for special purposes: A handbook for teachers of adult literacy. Macquarie University Press.

Hermansson, C., Jonsson, B., Levlin, M., Lindhé, A., Lundgren, B., & Norlund Shaswar, A. (2019). The (non)effect of joint construction in a genre-based approach to teaching writing. The Journal of Educational Research, 112(4), 483-494. https://doi.org/10.1080/00220671.2018.1563038

Hughes, A. (2008). Testing for language learners. Cambridge University.

Hyland, K. (2007). Genre pedagogy: Language, literacy, and L2 writing instruction. Journal of Second Language Writing, 16(3), 148-164. https://doi.org/10.1016/j.jslw.2007.07.005

Inbar-Lourie, O. (2008). Constructing a language assessment knowledge base: A focus on language assessment courses. Language Testing, 25(3), 385-402. https://doi.org/10.1177/0265532208090158

Ing, L. M., Musah, M. B., Al-Hudawi, S. H. V., Tahir, L. M., & Kamil, N. M. (2015). Validity of teacher made assessment: A table of specification approach. Asian Social Science, 11(5), 193-200. http://dx.doi.org/10.5539/ass.v11n5p193

Janatifar, M., & Marandi, S. S. (2018). Iranian EFL teachers’ language assessment literacy (LAL) under an assessing lens. Applied Research on the English Language, 7(3), 361-382. https://doi.org/10.22108/are.2018.108818.1221

Jannati, S. (2015). ELT teachers’ language assessment literacy: Perceptions and practices. The International Journal of Research in Teacher Education, 6(2), 26-37.

Jeong, H. (2017). Narrative and expository genre effects on students, raters, and performance criteria. Assessing writing, 31, 113-125. https://doi.org/10.1016/j.asw.2016.08.006

Lee, I. (2017). Classroom writing assessment and feedback in L2 school contexts. Springer Singapore.

Leong, W. S., Ismail, H., Costa, J. S., & Tan, H. B. (2018). Assessment for learning research in East Asian countries. Studies in Educational Evaluation, 59, 270-277. https://doi.org/10.1016/j.stueduc.2018.09.005

Livingston, S. A. (2018). Test reliability-basic concepts [Research Memorandum ETS RM-18-01]. Educational Testing Service.

Minda, M. H. (2018). Content validity of EFL teacher-made assessment: The case of communicative English skills course at Ambo University. East African Journal of Social Sciences and Humanities, 3(1), 41-62. https://doi.org/10.20372/eajssh.v3i1.470

Mukminatien, N., Yaniafari, R. P., El Khoiri, N., & Ariyanto, M. S. A. (2021). The rater reliability of an argumentative writing assessment using genre-based scoring rubric. In N. Hayati, Suharyadi, S. Andreani, & U. P. Astuti (Eds.), The changing face of ELT: A festschrift for Prof. Ali Saukah and Prof. M. Adnan Latief (pp. 175-191). Universitas Negeri Malang.

Nation, I. S. P., & Macalister, J. (2010). Language curriculum design. Routledge.

Oluwatayo, J. (2012). Validity and reliability issues in educational research. Journal of Educational and Social Research 2(2), 391-400.

O’Neill, P., & Murphy, S. (2017). Postsecondary writing assessment. In C. Secolsky & D. B. Denison (Eds.), Handbook on measurement, assessment, and evaluation in higher education (pp. 586-604). Routledge.

Oshima, A., & Hogue, A. (1998). Writing academic English. Longman.

Oshima, A., & Hogue, A. (2007). Introduction to academic writing. Pearson Education.

Pallant, J. (2011). A step-by-step guide to data analysis using the SPSS program: Survival manual. Open University Press.

Pesiwarissa, R., Anaktototy, K., & Maruanaya, H. J. (2021). An analysis of EFL teacher-made tests. Matai International Journal of Language Education, 2(1), 1-13. https://doi.org/10.30598/matail.v2i1.5485

Pill, J., & Harding, L. (2013). Defining the language assessment literacy gap: Evidence from a parliamentary inquiry. Language Testing, 30(3), 381-402. https://doi.org/10.1177/0265532213480337

Popham, W. J. (2009). Assessment literacy for teachers: Faddish or fundamental? Theory Into Practice, 48(1), 4-11. https://doi.org/10.1080/00405840802577536

Razavipour, K., Riazi, A., & Rashidi, N. (2011). On the interaction of test washback and teacher assessment literacy: The case of Iranian EFL secondary school teachers. English Language Teaching, 4(1), 156-161. https://doi.org/10.5539/elt.v4n1p156

Robson, C. (2011). Real world research: A resource for users of social research methods in applied settings (2nd ed.). John Wiley and Sons Ltd.

Santi, E., Setiawan, M. H., & Khumaedi, M. (2019). Developing assessment instrument on writing on descriptive paragraph based on picture of junior high school students. Journal of Research and Educational Research Evaluation, 8(2), 116-122. https://doi.org/10.15294/jere.v8i2.36086

Schedl, M., & Malloy, J. (2014). Writing items and tasks. In A. J. Kunnan (Ed.), The companion to language assessment (pp. 796-813). Wiley. https://doi.org/10.1002/9781118411360.wbcla025

Setiabudi, A., Mulyadi, M., & Puspita, H. (2019). An analysis of validity and reliability of a teacher-made test. Journal of English Education and Teaching, 3(4), 522-532. https://doi.org/10.33369/jeet.3.4.522-532

Shepard, L. A. (2000, February). The role of classroom assessment in teaching and learning [CSE Technical Report 517]. National Center Research on Evaluation, Standards, and Student Testing University of California.

Smalley, R. L., Ruetten, M. K., & Kozyrev, J. R. (2001). Refining composition skills. Rhetoric and grammar. Heinle & Heinle.

Stiggins, R. (1991). Assessment literacy. Phi Delta Kappan, 72(7), 534-539. https://www.jstor.org/stable/20404455

Stiggins, R. (2002). Assessment crisis: The absence of assessment for learning. Phi Delta Kappan Magazine, 83(10), 758-765. https://doi.org/10.1177/003172170208301010

Suharyadi, S. (2021). Genre and text type. In N. Hayati, Suharyadi, S. Andreani, & U. P. Astuti (Eds.), The changing face of ELT: A festschrift for Prof. Ali Saukah and Prof. M. Adnan Latief (pp. 135-147). Universitas Negeri Malang.

Tsagari, D., & Vogt, K. (2017). Assessment literacy of foreign language teachers around Europe: Research, challenges, and prospects. Language Testing and Assessment, 6(1), 41-63. https://doi.org/10.58379/UHIX9883

Ugwu, N. G., & Mkpuma, S. O. (2019). Ensuring quality in education: Validity in teacher-made language tests in secondary schools in Ebonyi state. American Journal of Educational Research, 7(7), 518-523.

Wu, X. M., Zhang, L. J., & Dixon, H. R. (2021). Implementing assessment for learning (AfL) in Chinese university EFL classes: Teachers’ values and practices. System, 101, 102589. https://doi.org/10.1016/j.system.2021.102589

Xu, Y., & Brown, G. T. L. (2016). Teacher assessment literacy in practice: A reconceptualization. Teaching and Teacher Education, 58, 149-162. https://doi.org/10.1016/j.tate.2016.05.010




DOI: https://doi.org/10.24815/siele.v11i1.26774

Article Metrics

Abstract view : 0 times
PDF - 0 times

Refbacks

  • There are currently no refbacks.


Print ISSN: 2355-2794, Online ISSN: 2461-0275

This work is licensed under a Creative Commons Attribution 4.0 International License.


View Journal Stats