Automatic determination of semantic similarity of student answers with the standard one using modern models

N. S. Lagutina ; K. V. Lagutina ; V. N. Kopnin

Modelirovanie i analiz informacionnyh sistem, Tome 31 (2024) no. 2, pp. 194-205.

Résumé

The paper presents the results of a study of modern text models in order to identify, on their basis, the semantic similarity of English-language texts. The task of determining semantic similarity of texts is an important component of many areas of natural language processing: machine translation, information retrieval, question and answer systems, artificial intelligence in education. The authors solved the problem of classifying the proximity of student answers to the teacher's standard answer. The neural network language models BERT and GPT, previously used to determine the semantic similarity of texts, the new neural network model Mamba, as well as stylometric features of the text were chosen for the study. Experiments were carried out with two text corpora: the Text Similarity corpus from open sources and the custom corpus, collected with the help of philologists. The quality of the problem solution was assessed by precision, recall, and F-measure. All neural network language models showed a similar F-measure quality of about 86% for the larger Text Similarity corpus and 50–56% for the custom corpus. A completely new result was the successful application of the Mamba model. However, the most interesting achievement was the use of vectors of stylometric features of the text, which showed 80% F-measure for the custom corpus and the same quality of problem solving as neural network models for another corpus.

Détail

Keywords: natural language processing, text similarity, neural network language models, assessing students' open responses, artificial intelligence in education.
Mots-clés : text classification

Bibliographie

[1] R. Gao, H. E. Merzdorf, S. Anwar, M. C. Hipwell, A. Srinivasa, “Automatic assessment of text-based responses in post-secondary education: A systematic review”, Computers and Education: Artificial Intelligence, 6 (2024), 100–206 | DOI

[2] J. Wang, Y. Dong, “Measurement of text similarity: A survey”, Information, 11:9 (2020), 421 | DOI

[3] A. Rozeva, S. Zerkova, “Assessing semantic similarity of texts-methods and algorithms”, AIP Conference Proceedings, 1910:1 (2017), 060–012 | DOI

[4] P. D. Wibisono, A. Asad, A. Chintan, “Short text similarity measurement methods: A review”, Soft Computing, 25 (2021), 4699–4723 | DOI

[5] N. S. Lagutina, M. V. Tihomirov, N. K. Mastakova, “Algoritm avtomaticheskogo postroeniya yazykovogo profilya uchashchegosya”, Zametki po informatike i matematike, 2023, no. 15, 58–65 (in Russian)

[6] O. B. Mishunin, A. P. Savinov, D. I. Firstov, “Sostoyanie i uroven' razrabotok sistem avtomaticheskoj ocenki svobodnyh otvetov na estestvennom yazyke”, Modern high technologies, 2016, no. 1, 38–44 (in Russian)

[7] L. Zahrotun, “Comparison Jaccard similarity, cosine similarity and combined both of the data clustering with Shared Nearest Neighbor method”, Computer Engineering and Applications Journal, 5:1 (2016), 11–18 | DOI

[8] H. A. Abdeljaber, “Automatic Arabic short answers scoring using longest common subsequence and Arabic WordNet”, IEEE Access, 9 (2021), 76-433–76-445 | DOI

[9] S. Sultana, I. Biskri, “Identifying similar sentences by using n-grams of characters”, Recent Trends and Future Technology in Applied Intelligence, Proceedings of 31st International Conference on Industrial Engineering and Other Applications of Applied Intelligent Systems, Springer, 2018, 833–843 | DOI

[10] S. Vij, D. Tayal, A. Jain, “A machine learning approach for automated evaluation of short answers using text similarity based on WordNet graphs”, Wireless Personal Communications, 111 (2020), 1271–1282 | DOI

[11] Y. Zhou, C. Li, G. Huang, Q. Guo, H. Li, X. Wei, “A short-text similarity model combining semantic and syntactic information”, Electronics, 12:14 (2023), 3126 | DOI

[12] M. Mohler, R. Mihalcea, “Text-to-text semantic similarity for automatic short answer grading”, Proceedings of the 12th Conference of the European Chapter of the ACL, EACL 2009, 2009, 567–575

[13] M. Han, X. Zhang, X. Yuan, J. Jiang, W. Yun, C. Gao, “A survey on the techniques, applications, and performance of short text semantic similarity”, Concurrency and Computation: Practice and Experience, 33:5 (2021), e5971 | DOI

[14] S. Roy, S. Dandapat, A. Nagesh, Y. Narahari, “Wisdom of students: A consistent automatic short answer grading technique”, Proceedings of the 13th International Conference on Natural Language Processing, 2016, 178–187

[15] A. Ahmed, A. Joorabchi, M. J. Hayes, “On deep learning approaches to automated assessment: Strategies for short answer grading”, Proceedings of the 14th International Conference on Computer Supported Education, v. 2, 2022, 85–94 | DOI

[16] A. Ahmed, A. Joorabchi, M. J. Hayes, “On the application of sentence transformers to automatic short answer grading in blended assessment”, Proceedings of the 33rd Irish Signals and Systems Conference (ISSC), IEEE, 2022, 1–6 | DOI

[17] L. Camus, A. Filighera, “Investigating transformers for automatic short answer grading”, Proceedings of the 21st International Conference Artificial Intelligence in Education, v. II, Springer, 2020, 43–48 | DOI

[18] D. Viji, S. Revathy, “A hybrid approach of weighted fine-tuned BERT extraction with deep Siamese Bi-LSTM model for semantic text similarity identification”, Multimedia Tools and Applications, 81:5 (2022), 6131–6157 | DOI

[19] D. Witschard, I. Jusufi, R. M. Martins, K. Kucher, A. Kerren, “Interactive optimization of embedding-based text similarity calculations”, Information Visualization, 21:4 (2022), 335–353 | DOI

[20] T. Brown et al, “Language models are few-shot learners”, Advances in neural information processing systems, 33 (2020), 1877–1901

[21] D. Shashavali et al, “Sentence similarity techniques for short vs variable length text using word embeddings”, Computacion y Sistemas, 23:3 (2019), 999–1004 | DOI

[22] B. Hassan, S. E. Abdelrahman, R. Bahgat, I. Farag, “UESTS: An unsupervised ensemble semantic textual similarity method”, IEEE Access, 7 (2019), 85-462–85-462 | DOI

[23] I. Gagliardi, M. T. Artese, “Ensemble-based short text similarity: An easy approach for multilingual datasets using transformers and wordnet in real-world scenarios”, Big Data and Cognitive Computing, 7:4 (2023), 158 | DOI

[24] N. Lagutina, K. Lagutina, A. Brederman, N. Kasatkina, “Text classification by CEFR levels using machine learning methods and BERT language model”, Modeling and Analysis of Information Systems, 30:3 (2023), 202–213 | DOI | MR

[25] P. Qi, Y. Zhang, Y. Zhang, J. Bolton, C. D. Manning, “Stanza: A Python natural language processing toolkit for many human languages”, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 2020, 101–108 | DOI | MR