Geodesic
Annotation of text corpora by sentiment and presence of irony within a project of citizen science
Modelirovanie i analiz informacionnyh sistem, Tome 30 (2023) no. 1, pp. 86-100.

Voir la notice de l'article provenant de la source Math-Net.Ru

The paper is devoted to construction of a sentence corpus annotated by the general sentiment into 4 classes (positive, negative, neutral, and mixed), a corpus of phrasemes annotated by the sentiment into 3 classes (positive, negative, and neutral), and a corpus of sentences annotated by the presence or absence of irony. The annotation was done by volunteers within the project “Prepare texts for algorithms” on the portal “People of science”. The existing knowledge on the domain regarding each task was the basis to develop guidelines for annotators. A technique of statistical analysis of the annotation result based on the distributions and agreement measures of the annotations performed by various annotators was also developed. For the annotation of sentences by irony and phrasemes by the sentiment the agreement measures were rather high (the full agreement rate of 0.60–0.99), whereas for the annotation of sentences by the general sentiment the agreement was low (the full agreement rate of 0.40), presumably, due to the higher complexity of the task. It was also shown that the results of automatic algorithms of detecting the sentiment of sentences improved by 12–13 % when using a corpus for which all the annotators (from 3 till 5) had the agreement, in comparison with a corpus annotated by only one volunteer.
Keywords: sentiment analysis, text corpus, statistical analysis, agreement measures, citizen science. 
@article{MAIS_2023_30_1_a5,
     author = {I. V. Paramonov and A. Yu. Poletaev},
     title = {Annotation of text corpora by sentiment and presence of irony within a project of citizen science},
     journal = {Modelirovanie i analiz informacionnyh sistem},
     pages = {86--100},
     publisher = {mathdoc},
     volume = {30},
     number = {1},
     year = {2023},
     language = {ru},
     url = {http://geodesic.mathdoc.fr/item/MAIS_2023_30_1_a5/}
}
TY  - JOUR
AU  - I. V. Paramonov
AU  - A. Yu. Poletaev
TI  - Annotation of text corpora by sentiment and presence of irony within a project of citizen science
JO  - Modelirovanie i analiz informacionnyh sistem
PY  - 2023
SP  - 86
EP  - 100
VL  - 30
IS  - 1
PB  - mathdoc
UR  - http://geodesic.mathdoc.fr/item/MAIS_2023_30_1_a5/
LA  - ru
ID  - MAIS_2023_30_1_a5
ER  - 
%0 Journal Article
%A I. V. Paramonov
%A A. Yu. Poletaev
%T Annotation of text corpora by sentiment and presence of irony within a project of citizen science
%J Modelirovanie i analiz informacionnyh sistem
%D 2023
%P 86-100
%V 30
%N 1
%I mathdoc
%U http://geodesic.mathdoc.fr/item/MAIS_2023_30_1_a5/
%G ru
%F MAIS_2023_30_1_a5
I. V. Paramonov; A. Yu. Poletaev. Annotation of text corpora by sentiment and presence of irony within a project of citizen science. Modelirovanie i analiz informacionnyh sistem, Tome 30 (2023) no. 1, pp. 86-100. http://geodesic.mathdoc.fr/item/MAIS_2023_30_1_a5/

[1] V. Masoumi, M. Salehi, H. Veisi, G. Haddadian, V. Ranjbar, and M. Sahebdel, TeleCrowd: A Crowdsourcing Approach to Create Informal to Formal Text Corpora, 2020, arXiv: 2004.11771 [cs.SI]

[2] E. Mitiagina, M. Borodataya, E. Volchenkova, N. Ershova, M. Luchinina, and E. Kotelnikov, “Russian Text Corpus of Intimate Partner Violence: Annotation Through Crowdsourcing”, 7th International Conference on Electronic Governance and Open Society: Challenges in Eurasia, EGOSE 2020, Springer, 2020, 306–321 | DOI

[3] S. Mohammad, “A practical guide to sentiment annotation: Challenges and solutions”, Proceedings of the 7th workshop on computational approaches to subjectivity, sentiment and social media analysis, Association for Computational Linguistics, 2016, 174–179 | DOI

[4] S. M. Mohammad, P. Sobhani, and S. Kiritchenko, “Stance and Sentiment in Tweets”, Special Section of the ACM Transactions on Internet Technology on Argumentation in Social Media, 17:3, 1–23 | MR

[5] B. R. Chakravarthi, V. Muralidaran, R. Priyadharshini, and J. P. McCrae, “Corpus Creation for Sentiment Analysis in Code-Mixed Tamil-English Text”, Proceedings of the 1st Joint SLTU and CCURL Workshop, SLTU-CCURL 2020, 2020, 202–210

[6] K. Krippendorff, Content analysis: an introduction to its methodology, SAGE Publications, Inc, Thousand Oaks, CA, 2013

[7] Y. Zhao, B. Qin, and T. Liu, “Creating a fine-grained corpus for chinese sentiment analysis”, IEEE Intelligent Systems, 30:1 (2014), 36–43 | DOI

[8] J. Cohen, “A coefficient of agreement for nominal scales”, Educational and psychological measurement, 20:1 (1960), 37–46 | DOI

[9] J. Bu et al., “ASAP: A Chinese Review Dataset Towards Aspect Category Sentiment Analysis and Rating Prediction”, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021, 2069–2079

[10] M. Navas-Loro, V. Rodríguez-Doncel, I. Santana-Perez, and A. S'anchez, “Spanish Corpus for Sentiment Analysis Towards Brands”, Speech and Computer, SPECOM 2017, 2017, 680–689 | DOI

[11] J. L. Fleiss, “Measuring nominal scale agreement among many raters”, Psychological bulletin, 76:5 (1971), 378 | DOI

[12] A. Rogers, A. Romanov, A. Rumshisky, S. Volkova, M. Gronas, and A. Gribov, “RuSentiment: An enriched sentiment analysis dataset for social media in Russian”, Proceedings of the 27th international conference on computational linguistics, 2018, 755–763

[13] T. V. Zherebilo, Slovar lingvisticheskih terminov, OOO Piligrim, Nazran, 2010

[14] K. Krippendorff, Computing Krippendorff's Alpha-Reliability, , 2008 (Accessed: Jan. 17, 2023) https://repository.upenn.edu/asc_papers/43/

[15] J. Hughes, “krippendorffsalpha: An R package for measuring agreement using Krippendorff's alpha coefficient”, The R Journal, 13:1 (2021), 413–425 | DOI

[16] L. A. Jeni, J. F. Cohn, and F. De La Torre, “Facing imbalanced data–recommendations for the use of performance metrics”, 2013 Humaine association conference on affective computing and intelligent interaction, 2013, 245–251 | MR

[17] A. Y. Poletaev and I. V. Paramonov, “Recursive sentiment detection algorithm for Russian sentences”, Modelirovanie i Analiz Informatsionnykh Sistem, 29:2 (2022), 134–147

[18] S. Smetanin and M. Komarov, “Deep transfer learning baselines for sentiment analysis in Russian”, Information Processing Management, 58:3 (2021), 102484 | DOI

[19] R. Artstein and M. Poesio, “Inter-coder agreement for computational linguistics”, Computational linguistics, 34:4 (2008), 555–596 | DOI