Readability formulas for three levels of Russian school textbooks

V. Solovyev ; V. Ivanov ; M. Solnyshkina

Zapiski Nauchnykh Seminarov POMI, Investigations on applied mathematics and informatics. Part II–1, Tome 529 (2023), pp. 140-156

Résumé

In this work, we propose a new text complexity formula aimed at assessing the complexity of Russian school textbooks. We used the annotated Russian Academic Corpus containing over 5 million tokens as the training and validation data and employed machine learning methods in the study. The values of 4 parameters in each of the 154 texts used for the research were measured with the help of the tools from the Spacy library. Comparative analysis of the new and existing complexity formulas suggests that the differences between them are indicative and the new formulas provide more accurate results. This research advances our understanding of the interdependency between frequency and text complexity and provides a framework for effective implementation of lexical frequency patterns in discourse complexity studies. The findings can be implemented by textbooks writers and test developers to select and modify texts for specific categories of readers. Other areas of application include website design, surveys, and semantic analysis of social networks.

Bibliographie

[1] S Bax, Text inspector, Online text analysis tool, 2012

[2] E. A. Betts, Foundations of reading instruction, with emphasis on differentiated guidance, 1946

[3] Meri Coleman and Ta Lin Liau, “A computer readability formula designed for machine scoring”, Journal of Applied Psychology, 60:2 (1975), 283

[4] S. A. Crossley, K. Kyle, D. S. McNamara, “The tool for the automatic analysis of text cohesion (taaco): Automatic assessment of local, global, and text cohesion”, Behavior research methods, 48 (2016), 1227–1237

[5] E. Dale, J. S. Chall, “A formula for predicting readability: Instructions”, Educational research bulletin, 1948, 37–54

[6] A. Davison, G. M. Green, Linguistic complexity and text comprehension: Readability issues reconsidered, Routledge, 1988

[7] C. Dragos, R. Stefan, D. Mihai, “Readerbench: Multilevel analysis of russian text characteristics”, Russian J. Linguistics, 26:2 (2022), 342–370

[8] W. H. DuBay, The classic readability studies, Online Submission, 2007

[9] A. A. Dyakova, “Ob'ektivnaja i sub'ektivnaja slozhnost' dokumentnyh tekstov”, Vestnik Volgogradskogo Gosudarstvennogo Univer., Ser. 2, Yazykoznanie, 1:30 (2016), 63–69

[10] J. S. B. Evans, Thinking twice: Two minds in one brain, Oxford University Press, 2010

[11] J. S. B. Evans, “Dual-process theories of reasoning: Contemporary issues and developmental applications”, Developmental Review, 31:2–3 (2011), 86–102

[12] R. Flesch, “A new readability yardstick.”, Journal Applied Psychology, 32:3 (1948), 221

[13] G. Gatiyatullina, M. Solnyshkina, V. Solovyev, A. Danilov, E. Martynova, I. Yarmakeev, “Computing russian morphological distribution patterns using rusac online server”, 2020 13th International Conference on Developments in eSystems Engineering (DeSE), IEEE, 2020, 393–398

[14] R. Gunning et al., Technique of clear writing, 1952

[15] V. V. Ivanov, M. I. Solnyshkina, V. D. Solovyev, “Efficiency of text readability features in russian academic texts”, Komp'juternaja Lingvistika I Intellektual'nye Tehnologii, 2018, 284–293

[16] P. Kendeou, P. Van den Broek, M. J. White, J. S. Lynch, “Predicting reading comprehension in early elementary school: The independent contributions of oral language and decoding skills.”, J. Educational Psychology, 101:4 (2009), 765

[17] G. R. Klare, P. P. Rowe, M. G. St. John, L. M. Stolurow, “Automation of the flesch reading ease readability formula, with various options”, Reading research quarterly, 1969, 550–559

[18] J. D. Kniffin, “The new readability requirements for military technical manuals”, Technical Communication, 1979, 16–19

[19] R. V. Kupriyanov, M. I. Solnyshkina, M. Dascalu, T. A. Soldatkina, “Lexical and syntactic features of academic russian texts: a discriminant analysis”, Research Result. Theoretical and Applied Linguistics, 8:4 (2022), 105–122

[20] K. Kyle, S. Crossley, C. Berger, “The tool for the automatic analysis of lexical sophistication (taales): version 2.0”, Behavior research methods, 50 (2018), 1030–1046

[21] A. N. Laposhina, M. Yu. Lebedeva, “Tekstometr: Online-instrument opredeleniya urovnya slozhnosti teksta po russkomu yazyku kak inostrannomu”, Rusistika, 19:3 (2021), 331–345 (Russ.) | DOI

[22] G. A. Lesskis, “O zavisimosti mezhdu razmerom predlozheniya i ego strukturoy v raznykh vidakh teksta”, Voprosy Yazykoznanija, 1964, no. 3, 99

[23] O. Lyashevskaya, S. Sharov, The frequency dictionary of modern Russian language, Azbukovnik, M., 2009

[24] K. S. MakKarti, D. S. MakNamara, M. I. Solnyshkina, F. Kh. Tarasova, R. V. Kupriyanov, “Testirovanie po russkomu yazyku: ocenka ponimaniya teksta”, Vestnik Volgogradskogo Gosudarstvennogo Univer. Ser. 2, Yazykoznanie, 18:3 (2019)

[25] M. S. Matskovskiy, “The problems of typed text readability”, Semantic Perception of Verbal Communication (in the Conditions of Mass Communication), IYa AN SSSR Publ, M., 1976, 126–142

[26] G. Harry Mc Laughlin, “Smog grading-a new readability formula”, J. Reading, 12:8 (1969), 639–646

[27] D. McNamara, A. Graesser, P. McCarthy, Zh. Cai, Automated evaluation of text and discourse with coh-metrix, Cambridge University Press, 2014

[28] V. Medvedev, M. Solnyshkina, “Technologies of assessing and enhancing cohesion of instructional and narrative texts”, Science and Global Challenges of the 21st Century-Science and Technology, Proceedings of the International Perm Forum “Science and Global Challenges of the 21st Century”, Springer, 2022, 693–712

[29] D. A. Morozov, A. V. Glazkova, B. L. Iomdin, “Text complexity and linguistic features: Their correlation in english and russian”, Russian J. Linguistics, 26:2 (2022), 426–448

[30] I. V. Oborneva, Automatic evaluation of the complexity of educational texts on the basis of statistical parameters, Ph.D. thesis, 2006

[31] M. Oliffe, E. Thompson, J. Johnston, D. Freeman, H. Bagga, P. K. K. Wong, “Assessing the readability and patient comprehension of rheumatology medicine information sheets: a cross-sectional health literacy study”, BMJ open, 9:2 (2019), e024582

[32] R. G. Piotrovskiy, K. B. Bektaiev, A. A. Piotrovskaya, Matematicheskaya lingvistika, Vy'sshaya shkola, M., 1977

[33] T. Shanahan, “Why children should be taught to read with more challenging texts”, Perspectives on Language and Literacy, 45:4 (2019), 17–19

[34] S. Sharoff, “What neural networks know about linguistic complexity”, Russian J. Linguistics, 26:2 (2022), 371–390

[35] Y. Shpakovskiy et al., Evaluation of the difficulty of perception and optimization of the text complexity, Ph.D. thesis, 2007

[36] M. Solnyshkina, V. Solovyev, E. Gafiyatova, E. Martynova, “Text complexity as interdisciplinary problem”, Voprosy Kognitivnoy Lingvistiki, 2022, 18–39

[37] M. Solnyshkina, V. Ivanov, V. Solovyev, “Readability formula for russian texts: a modified version”, Advances in Computational Intelligence, 17th Mexican International Conference on Artificial Intelligence, MICAI 2018, Proceedings (Guadalajara, Mexico, October 22–27, 2018), v. II, Springer, 2018, 132–145

[38] M. Solnyshkina, V. Solovyev, V. Ivanov, A. Danilov, “Studying text complexity in russian academic corpus with multi-level annotation”, Academy of Sciences of the Republic of Tatarstan, 126 (2018)

[39] M. I. Solnyshkina, G. M. Gatiyatullina, R. V. Kupriyanov, C. R. Ziganshina, “Lexical density as a complexity predictor: the case of science and social studies textbooks”, RESEARCH RESULT Theoretical and Applied Linguistics, 2023

[40] M. I. Solnyshkina, D. S. McNamara, R. R. Zamaletdinov, et al., “Natural language processing and discourse complexity studies”, Russian J. Linguistics, 26:2 (2022), 317–341

[41] V. Solovyev, V. Ivanov, M. Solnyshkina, “Assessment of reading difficulty levels in russian academic texts: Approaches and metrics”, J. Intelligent Fuzzy Systems, 34:5 (2018), 3049–3058

[42] V. Solovyev, M. Solnyshkina, V. Ivanov, S. Timoshenko, “Complexity of russian academic texts as the function of syntactic parameters”, Computational Linguistics and Intelligent Text Processing, 19th International Conference, CICLing 2018, Revised Selected Papers (Hanoi, Vietnam, March 18–24, 2018), v. I, Springer, 2023, 168–179

[43] V. D. Solovyev, M. Islamov, M. Solnyshkina, R. Kupriyanov, E. Gafiyatova, “Sentiment analysis for russian academic texts: A lexicon-based approach”, IMS, 2021, 89–97

[44] V. D. Solovyev, M. I. Solnyshkina, D. S. McNamara, et al., “Computational linguistics and discourse complexology: Paradigms and research methods”, Russian J. Linguistics, 26:2 (2022), 275–316

[45] Z. Wang, T. O'Reilly, J. Sabatini, K. S. McCarthy, D. S. McNamara, “A tale of two tests: The role of topic and general academic knowledge in traditional versus contemporary scenario-based reading”, Learning and Instruction, 73 (1014), 101462