Geodesic
A model for automated business writing assessment
Modelirovanie i analiz informacionnyh sistem, Tome 29 (2022) no. 4, pp. 348-365

Voir la notice de l'article provenant de la source Math-Net.Ru

This study is aimed at building an automated model for business writing assessment, based on 14 rubrics that integrate EFL teacher assessment frameworks and identify expected performance against various criteria (including language, task fulfillment, content knowledge, register, format, and cohesion). We developed algorithms for determining the corresponding numerical features using methods and tools for automatic text analysis. The algorithms are based on a syntactic analysis with the use of dictionaries. The model performance was subsequently evaluated on a corpus of 20 teacher-assessed business letters. Heat maps and UMAP results represent comparison between teachers' and automated score reports. Results showed no significant discrepancies between teachers' and automated score reports, yet detected bias in teachers' reports. Findings suggest that the developed model has proved to be an efficient tool for natural language processing with high interpretability of the results, the roadmap for further improvement and a valid and unbiased alternative to teachers' assessment. The results may lay the groundwork for developing an automatic students' language profile. Although the model was specifically designed for business letter assessment, it can be easily adapted for assessing other writing tasks, e.g. by replacing dictionaries.
Keywords: natural language processing, text features, automated essay scoring, business letter. 
@article{MAIS_2022_29_4_a3,
     author = {D. D. Zafievsky and N. S. Lagutina and O. A. Melnikova and A. Y. Poletaev},
     title = {A model for automated business writing assessment},
     journal = {Modelirovanie i analiz informacionnyh sistem},
     pages = {348--365},
     publisher = {mathdoc},
     volume = {29},
     number = {4},
     year = {2022},
     language = {ru},
     url = {http://geodesic.mathdoc.fr/item/MAIS_2022_29_4_a3/}
}
TY  - JOUR
AU  - D. D. Zafievsky
AU  - N. S. Lagutina
AU  - O. A. Melnikova
AU  - A. Y. Poletaev
TI  - A model for automated business writing assessment
JO  - Modelirovanie i analiz informacionnyh sistem
PY  - 2022
SP  - 348
EP  - 365
VL  - 29
IS  - 4
PB  - mathdoc
UR  - http://geodesic.mathdoc.fr/item/MAIS_2022_29_4_a3/
LA  - ru
ID  - MAIS_2022_29_4_a3
ER  - 
%0 Journal Article
%A D. D. Zafievsky
%A N. S. Lagutina
%A O. A. Melnikova
%A A. Y. Poletaev
%T A model for automated business writing assessment
%J Modelirovanie i analiz informacionnyh sistem
%D 2022
%P 348-365
%V 29
%N 4
%I mathdoc
%U http://geodesic.mathdoc.fr/item/MAIS_2022_29_4_a3/
%G ru
%F MAIS_2022_29_4_a3
D. D. Zafievsky; N. S. Lagutina; O. A. Melnikova; A. Y. Poletaev. A model for automated business writing assessment. Modelirovanie i analiz informacionnyh sistem, Tome 29 (2022) no. 4, pp. 348-365. http://geodesic.mathdoc.fr/item/MAIS_2022_29_4_a3/