Geodesic
Text classification by CEFR levels using machine learning methods and BERT language model
Modelirovanie i analiz informacionnyh sistem, Tome 30 (2023) no. 3, pp. 202-213.

Voir la notice de l'article provenant de la source Math-Net.Ru

This paper presents a study of the problem of automatic classification of short coherent texts (essays) in English according to the levels of the international CEFR scale. Determining the level of text in natural language is an important component of assessing students knowledge, including checking open tasks in e-learning systems. To solve this problem, vector text models were considered based on stylometric numerical features of the character, word, sentence structure levels. The classification of the obtained vectors was carried out by standard machine learning classifiers. The article presents the results of the three most successful ones: Support Vector Classifier, Stochastic Gradient Descent Classifier, LogisticRegression. Precision, recall and F-score served as quality measures. Two open text corpora, CEFR Levelled English Texts and BEA-2019, were chosen for the experiments. The best classification results for six CEFR levels and sublevels from A1 to C2 were shown by the Support Vector Classifier with F-score 67 % for the CEFR Levelled English Texts. This approach was compared with the application of the BERT language model (six different variants). The best model, bert-base-cased, provided the F-score value of 69 %. The analysis of classification errors showed that most of them are between neighboring levels, which is quite understandable from the point of view of the domain. In addition, the quality of classification strongly depended on the text corpus, that demonstrated a significant difference in F-scores during application of the same text models for different corpora. In general, the obtained results showed the effectiveness of automatic text level detection and the possibility of its practical application.
Keywords: natural language processing, CEFR
Mots-clés : text classification, BERT. 
@article{MAIS_2023_30_3_a1,
     author = {N. S. Lagutina and K. V. Lagutina and A. M. Brederman and N. N. Kasatkina},
     title = {Text classification by {CEFR} levels using machine learning methods and {BERT} language model},
     journal = {Modelirovanie i analiz informacionnyh sistem},
     pages = {202--213},
     publisher = {mathdoc},
     volume = {30},
     number = {3},
     year = {2023},
     language = {ru},
     url = {http://geodesic.mathdoc.fr/item/MAIS_2023_30_3_a1/}
}
TY  - JOUR
AU  - N. S. Lagutina
AU  - K. V. Lagutina
AU  - A. M. Brederman
AU  - N. N. Kasatkina
TI  - Text classification by CEFR levels using machine learning methods and BERT language model
JO  - Modelirovanie i analiz informacionnyh sistem
PY  - 2023
SP  - 202
EP  - 213
VL  - 30
IS  - 3
PB  - mathdoc
UR  - http://geodesic.mathdoc.fr/item/MAIS_2023_30_3_a1/
LA  - ru
ID  - MAIS_2023_30_3_a1
ER  - 
%0 Journal Article
%A N. S. Lagutina
%A K. V. Lagutina
%A A. M. Brederman
%A N. N. Kasatkina
%T Text classification by CEFR levels using machine learning methods and BERT language model
%J Modelirovanie i analiz informacionnyh sistem
%D 2023
%P 202-213
%V 30
%N 3
%I mathdoc
%U http://geodesic.mathdoc.fr/item/MAIS_2023_30_3_a1/
%G ru
%F MAIS_2023_30_3_a1
N. S. Lagutina; K. V. Lagutina; A. M. Brederman; N. N. Kasatkina. Text classification by CEFR levels using machine learning methods and BERT language model. Modelirovanie i analiz informacionnyh sistem, Tome 30 (2023) no. 3, pp. 202-213. http://geodesic.mathdoc.fr/item/MAIS_2023_30_3_a1/

[1] E. del Gobbo, A. Guarino, B. Cafarelli, L. Grilli, P. Limone, “Automatic evaluation of open-ended questions for online learning. A systematic mapping”, Studies in Educational Evaluation, 77, Elsevier, 2023, 101258 | DOI

[2] N. Galichev, P. Shirogorodskaya, “Problema avtomaticheskogo izmereniya slozhnyh konstruktov cherez otkrytye zadaniya”, Hxi mezhdunarodnaya nauchno-prakticheskaya konferenciya molodyh issledovatelej obrazovaniya, Novosibirskij gosudarstvennyj pedagogicheskij universitet, 2022, 695–697 (in Russian)

[3] L. E. Adamova, O. Surikova, I. G. Bulatova, O. O. Varlamov, “Application of the mivar expert system to evaluate the complexity of texts”, News of the Kabardin-Balkar scientific center of RAS, 2021, no. 2, 11–29, Kabardino-Balkarian Scientific Center of the Russian Academy of Sciences

[4] D. Ramesh, S. K. Sanampudi, “An automated essay scoring systems: a systematic literature review”, Artificial Intelligence Review, 55:3 (2022), 2495–2527 | DOI | MR

[5] K. P. Yancey, G. Laflair, A. Verardi, J. Burstein, “Rating short L2 essays on the CEFR scale with GPT-4”, Proceedings of the 18th workshop on innovative use of nlp for building educational applications (bea 2023), 2023, 576–584 | DOI

[6] A. Gasparetto, M. Marcuzzo, A. Zangari, A. Albarelli, “A survey on text classification algorithms: From text to predictions”, Information, 13:2 (2022), 83, MDPI | DOI

[7] V. Ramnarain-Seetohul, V. Bassoo, Y. Rosunally, “Similarity measures in automated essay scoring systems: a ten-year review”, Education and Information Technologies, 27:4 (2022), 5573–5604 | DOI

[8] Y. Yang, J. Zhong, “Automated essay scoring via example-based learning”, Web engineering, Springer, 2021, 201–208 | DOI

[9] N. N. Mikheeva, E. V. Shulyndina, “Features of training written Internet communication in a non-linguistic university”, Tambov University Review. Series: Humanities, 28:2 (2023), 405–414 | DOI

[10] V. J. Schmalz, A. Brutti, “Automatic assessment of English CEFR levels using BERT embeddings”, Proceedings of the eighth italian conference on computational linguistics, CEUR Workshop Proceedings, 2021

[11] Y. Arase, S. Uchida, T. Kajiwara, “CEFR-based sentence difficulty annotation and assessment”, Proceedings of the 2022 conference on empirical methods in natural language processing, 2022, 6206–6219 | DOI

[12] R. Jalota, P. Bourgonje, J. Van Sas, H. Huang, “Mitigating learnerese effects for CEFR classification”, Proceedings of the 17th workshop on innovative use of nlp for building educational applications (BEA 2022), 2022, 14–21 | DOI

[13] T. Gaillat, A. Simpkin, N. Ballier, B. Stearns, A. Sousa, M. Bouyé, M. Zarrouk, “Predicting CEFR levels in learners of English: the use of microsystem criterial features in a machine learning approach”, ReCALL, 34:2 (2022), 130–146 | DOI

[14] E. Kerz, D. Wiechmann, Y. Qiao, E. Tseng, M. Ströbel, “Automated classification of written proficiency levels on the CEFR-scale through complexity contours and RNNs”, Proceedings of the 16th workshop on innovative use of nlp for building educational applications, 2021, 199–209

[15] P. Yang, L. Li, F. Luo, T. Liu, X. Sun, “Enhancing topic-to-essay generation with external commonsense knowledge”, Proceedings of the 57th annual meeting of the association for computational linguistics, 2019, 2002–2012 | DOI

[16] E. Mayfield, A. W. Black, Should you fine-tune BERT for automated essay scoring?, Proceedings of the fifteenth workshop on innovative use of nlp for building educational applications, 2020, 151–162 | DOI

[17] J. M. Imperial, “BERT embeddings for automatic readability assessment”, Proceedings of the international conference on recent advances in natural language processing (ranlp 2021), 2021, 611–618 | DOI

[18] C. Bryant, M. Felice, Ø E. Andersen, T. Briscoe, “The BEA-2019 shared task on grammatical error correction”, Proceedings of the fourteenth workshop on innovative use of nlp for building educational applications, Association for Computational Linguistics, 2019, 52–75 | DOI

[19] K. V. Lagutina, A. M. Manakhova, “Automated search and analysis of the stylometric features that describe the style of the prose of 19th–21st centuries”, Automatic Control and Computer Sciences, 55:7 (2021), 866–876, Springer | DOI | MR

[20] A. M. Manakhova, N. S. Lagutina, “Analysis of the impact of the stylometric characteristics of different levels for the verification of authors of the prose”, Modeling and Analysis of Information Systems, 28:3 (2021), 260–279 (in Russian) | DOI | MR

[21] J. Devlin, M. Chang, K. Lee, K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding”, Proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, v. 1, 2019, 4171–4186 | MR

[22] V. Sanh, L. Debut, J. Chaumond, T. Wolf, DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter, 2020, arXiv: 1910.01108