Geodesic
A model for automated business writing assessment
Modelirovanie i analiz informacionnyh sistem, Tome 29 (2022) no. 4, pp. 348-365.

Voir la notice de l'article provenant de la source Math-Net.Ru

This study is aimed at building an automated model for business writing assessment, based on 14 rubrics that integrate EFL teacher assessment frameworks and identify expected performance against various criteria (including language, task fulfillment, content knowledge, register, format, and cohesion). We developed algorithms for determining the corresponding numerical features using methods and tools for automatic text analysis. The algorithms are based on a syntactic analysis with the use of dictionaries. The model performance was subsequently evaluated on a corpus of 20 teacher-assessed business letters. Heat maps and UMAP results represent comparison between teachers' and automated score reports. Results showed no significant discrepancies between teachers' and automated score reports, yet detected bias in teachers' reports. Findings suggest that the developed model has proved to be an efficient tool for natural language processing with high interpretability of the results, the roadmap for further improvement and a valid and unbiased alternative to teachers' assessment. The results may lay the groundwork for developing an automatic students' language profile. Although the model was specifically designed for business letter assessment, it can be easily adapted for assessing other writing tasks, e.g. by replacing dictionaries.
Keywords: natural language processing, text features, automated essay scoring, business letter. 
@article{MAIS_2022_29_4_a3,
     author = {D. D. Zafievsky and N. S. Lagutina and O. A. Melnikova and A. Y. Poletaev},
     title = {A model for automated business writing assessment},
     journal = {Modelirovanie i analiz informacionnyh sistem},
     pages = {348--365},
     publisher = {mathdoc},
     volume = {29},
     number = {4},
     year = {2022},
     language = {ru},
     url = {http://geodesic.mathdoc.fr/item/MAIS_2022_29_4_a3/}
}
TY  - JOUR
AU  - D. D. Zafievsky
AU  - N. S. Lagutina
AU  - O. A. Melnikova
AU  - A. Y. Poletaev
TI  - A model for automated business writing assessment
JO  - Modelirovanie i analiz informacionnyh sistem
PY  - 2022
SP  - 348
EP  - 365
VL  - 29
IS  - 4
PB  - mathdoc
UR  - http://geodesic.mathdoc.fr/item/MAIS_2022_29_4_a3/
LA  - ru
ID  - MAIS_2022_29_4_a3
ER  - 
%0 Journal Article
%A D. D. Zafievsky
%A N. S. Lagutina
%A O. A. Melnikova
%A A. Y. Poletaev
%T A model for automated business writing assessment
%J Modelirovanie i analiz informacionnyh sistem
%D 2022
%P 348-365
%V 29
%N 4
%I mathdoc
%U http://geodesic.mathdoc.fr/item/MAIS_2022_29_4_a3/
%G ru
%F MAIS_2022_29_4_a3
D. D. Zafievsky; N. S. Lagutina; O. A. Melnikova; A. Y. Poletaev. A model for automated business writing assessment. Modelirovanie i analiz informacionnyh sistem, Tome 29 (2022) no. 4, pp. 348-365. http://geodesic.mathdoc.fr/item/MAIS_2022_29_4_a3/

[1] A. Al-Bargi, “Exploring online writing assessment amid Covid-19: challenges and opportunities from teachers’ perspectives”, Arab World English Journal, 2022, 3–21, SocArXiv

[2] N. P. Soboleva, M. A. Nilova, “Obuchenie pis'mu studentov gumanitarnyh special'nostej s ispol'zovaniem sovremennyh obrazovatel'nyh tekhnologij”, Kazanskij vestnik molodyh uchyonyh, 2:5 (8) (2018), 57–59

[3] M. Fareed, A. Ashraf, M. Bilal, “Esl learners’ writing skills: problems, factors and suggestions”, Journal of education and social sciences, 4:2 (2016), 81–92

[4] K. N. A. Al-Mwzaiji, A. A. F. Alzubi, “Online self-evaluation: the EFL writing skills in focus”, Asian-Pacific Journal of Second and Foreign Language Education, 7:1 (2022), 1–16, Springer

[5] M. A. Hussein, H. Hassan, M. Nassef, “Automated language essay scoring systems: a literature review”, PeerJ Computer Science, 5 (2019), e208, PeerJ Inc.

[6] H. John Bernardin, S. Thomason, M. Ronald Buckley, J. S. Kane, “Rater rating-level bias and accuracy in performance appraisals: the impact of rater personality, performance management competence, and rater accountability”, Human Resource Management, 55:2 (2016), 321–340, Wiley Online Library

[7] Z. Ke, V. Ng, “Automated essay scoring: a survey of the state of the art”, Ijcai, 19 (2019), 6300–6308

[8] M. Uto, “A review of deep-neural automated essay scoring models”, Behaviormetrika, 48:2 (2021), 459–484, Springer

[9] S. Vajjala, “Automated assessment of non-native learner essays: investigating the role of linguistic features”, International Journal of Artificial Intelligence in Education, 28:1 (2018), 79–105, Springer

[10] K. Taghipour, H. T. Ng, “A neural approach to automated essay scoring”, Proceedings of the 2016 conference on empirical methods in natural language processing, 2016, 1882–1891

[11] L. Xia, J. Liu, Z. Zhang, “Automatic essay scoring model based on two-layer bi-directional long-short term memory network”, Proceedings of the 2019 3rd international conference on computer science and artificial intelligence, 2019, 133–137

[12] M. Uto, M. Okano, “Robust neural automated essay scoring using item response theory”, International conference on artificial intelligence in education, Springer, 2020, 549–561

[13] Y. Tay, M. Phan, L. A. Tuan, S. C. Hui, “Skipflow: incorporating neural coherence features for end-to-end automatic text scoring”, Proceedings of the AAAI conference on artificial intelligence, 32:1 (2018), 5948–5955

[14] Y. Farag, H. Yannakoudakis, T. Briscoe, “Neural automated essay scoring and coherence modeling for adversarially crafted input”, Proceedings of the 2018 conference of the north american chapter of the association for computational linguistics: human language technologies, Association for Computational Linguistics (ACL), 2018, 263–271

[15] Y. Yang, J. Zhong, “Automated essay scoring via example-based learning”, International conference on web engineering, Springer, 2021, 201–208

[16] E. Mayfield, A. W. Black, Should you fine-tune bert for automated essay scoring?, Proceedings of the fifteenth workshop on innovative use of NLP for building educational applications, 2020, 151–162

[17] R. Yang, J. Cao, Z. Wen, Y. Wu, X. He, “Enhancing automated essay scoring performance via fine-tuning pre-trained language models with combination of regression and ranking”, Findings of the association for computational linguistics: EMNLP 2020, 2020, 1560–1569

[18] M. Uto, Y. Xie, M. Ueno, “Neural automated essay scoring incorporating handcrafted features”, Proceedings of the 28th international conference on computational linguistics, 2020, 6077–6088

[19] I. Aomi, E. Tsutsumi, M. Uto, M. Ueno, “Integration of automated essay scoring models using item response theory”, International conference on artificial intelligence in education, Springer, 2021, 54–59

[20] W. Zhu, Y. Sun, “Automated essay scoring system using multi-model machine learning”, CS IT Conference Proceedings, 10:12 (2020), 109–117

[21] S. M. Darwish, S. K. Mohamed, “Automated essay evaluation based on fusion of fuzzy ontology and latent semantic analysis”, International conference on advanced machine learning technologies and applications, Springer, 2019, 566–575

[22] Y. Salim, V. Stevanus, E. Barlian, A. C. Sari, D. Suhartono, “Automated english digital essay grader using machine learning”, 2019 ieee international conference on engineering, technology and education, TALE, IEEE, 2019, 1–6

[23] R. Wilkens, D. Seibert, X. Wang, T. Fran{ç}ois, “Mwe for essay scoring english as a foreign language”, 2nd workshop on tools and resources for reading difficulties (READI), 2022, 62–69

[24] D. Ramesh, S. K. Sanampudi, “An automated essay scoring systems: a systematic literature review”, Artificial Intelligence Review, Springer, 2021, 1–33