Geodesic
Textual trends detection at OK
Vestnik Sankt-Peterburgskogo universiteta. Prikladnaâ matematika, informatika, processy upravleniâ, Tome 13 (2017) no. 3, pp. 313-325 Cet article a éte moissonné depuis la source Math-Net.Ru

Voir la notice de l'article

Social networks now serve not as a mere medium for entertainment, but as an information distribution channel that is replacing classical mass media. In this article we describe a scalable trend detection system implemented with the social network OK. Actors (users and communities) of social networks form a broad agenda. The content of social networks is specific: UGC (user generated content) is difficult to process; actors generate a multilingual text. This requires attracting a large number of highly paid professionals in the case of classical media analysis; modern social networks comprise a highly-connected society with high “response time”. Therefore, the system must work in real time; social networks are used by spammers as a platform for promotion and obtrusive advertising, therefore the system should contain the ability to filter spam content. Applying standard methods of media analysis to this seems impossible. It creates a natural demand for developing and implementing textual trend detection and analysis software. There are two main approaches of trend detection in academic papers: topic modeling (and further topics evolutionary analysis) and distributive models based on frequency-like properties of distinct terms. We conducted an analysis of scientific papers using both approaches taking into account the specific features of social networks. As a result of research, it was decided to use distributive models as a base for the system development. OK is one of the largest social networks in Russia and the CIS countries. Actors generate over 100M symbols of text every day. Even basic processing is a serious technical problem. So we are forced to use Big Data approaches through the development. We introduce lambda-architecture based on three main components: daily-batch processing component, based on Apache Spark; streaming processing component, based on Apache Samza; mini-batch processing component, based on Spark Streaming. The article describes in detail the architecture and technical features of each component. In conclusion we present the results of operating the system as well as discuss areas for further research and development. Refs 13. Figs 7. Table 1.
Keywords: natural language processing, trend detection, big data. 
@article{VSPUI_2017_13_3_a7,
     author = {E. A. Malyutin and D. Yu. Bugaichenko and A. N. Mishenin},
     title = {Textual trends detection at {OK}},
     journal = {Vestnik Sankt-Peterburgskogo universiteta. Prikladna\^a matematika, informatika, processy upravleni\^a},
     pages = {313--325},
     year = {2017},
     volume = {13},
     number = {3},
     language = {ru},
     url = {http://geodesic.mathdoc.fr/item/VSPUI_2017_13_3_a7/}
}
TY  - JOUR
AU  - E. A. Malyutin
AU  - D. Yu. Bugaichenko
AU  - A. N. Mishenin
TI  - Textual trends detection at OK
JO  - Vestnik Sankt-Peterburgskogo universiteta. Prikladnaâ matematika, informatika, processy upravleniâ
PY  - 2017
SP  - 313
EP  - 325
VL  - 13
IS  - 3
UR  - http://geodesic.mathdoc.fr/item/VSPUI_2017_13_3_a7/
LA  - ru
ID  - VSPUI_2017_13_3_a7
ER  - 
%0 Journal Article
%A E. A. Malyutin
%A D. Yu. Bugaichenko
%A A. N. Mishenin
%T Textual trends detection at OK
%J Vestnik Sankt-Peterburgskogo universiteta. Prikladnaâ matematika, informatika, processy upravleniâ
%D 2017
%P 313-325
%V 13
%N 3
%U http://geodesic.mathdoc.fr/item/VSPUI_2017_13_3_a7/
%G ru
%F VSPUI_2017_13_3_a7
E. A. Malyutin; D. Yu. Bugaichenko; A. N. Mishenin. Textual trends detection at OK. Vestnik Sankt-Peterburgskogo universiteta. Prikladnaâ matematika, informatika, processy upravleniâ, Tome 13 (2017) no. 3, pp. 313-325. http://geodesic.mathdoc.fr/item/VSPUI_2017_13_3_a7/

[1] Lau J. H., Collier N., Baldwin T., “On-line trend analysis with topic models: twitter trends detection topic model online”, Proceedings of COLING, Technical papers (Mumbai, 2012), 1519–1534

[2] Ahmed A., Xing E. P., “Timeline: A dynamic hierarchical dirichlet process model for recovering birth/death and evolution of topics in text stream”, Proceedings of the Twenty-Sixth Conference on Uncertainty in Artificial Intelligence, 2010, 20–29

[3] Schubert E., Weiler M., Kriegel H.-P., “Signitrend: scalable detection of emerging topics in textual streams by hashed significance thresholds”, Proceedings of the 20th ACM SIGKDD International conference on Knowledge discovery and data mining (2014), 871–880

[4] Cvijikj I. P., Michahelles F., “Monitoring trends on facebook”, IEEE Ninth Intern. Conference on Dependable, Autonomic and Secure Computing (DASC) (2011), 895–902

[5] Finch T., Incremental calculation of weighted mean and variance, Technical report No 4, Cambridge, 2009

[6] Ester M., Kriegel H.-P., Sander J., Xu X., “A density-based algorithm for discovering clusters in large spatial databases with noise”, Kdd, 96:34 (1996), 226–231

[7] Open-source library for language detection, (accessed: 26.02.2017) https://github.com/optimaize/language-detector

[8] Additional language profile for CIS-languages, (accessed: 26.02.2017) https://github.com/denniean/language_profiles

[9] Jeffrey D., Ullman Anand Rajaraman, Jure Leskovec, Mining of massive datasets, , 2013 (accessed: 26.02.2017) http://infolab.stanford.edu/ũllman/mmds.html

[10] Open-source library for data analysis, (accessed: 26.02.2017) https://elki-project.github.io/

[11] Scalable stream processing platform, (accessed: 26.02.2017) https://kafka.apache.org/

[12] Apache Samza: distributed stream processing framework, (accessed: 26.02.2017) http://samza.apache.org/

[13] Apache Zeppelin: web-dashboard for interactive data analysis, (accessed: 26.02.2017) https://zeppelin.apache.org/