Geodesic
Improved sentiment classification of images using a two-stage relieff and modified SCA feature selection
Nečetkie sistemy i mâgkie vyčisleniâ, Tome 19 (2024) no. 2, pp. 53-87.

Voir la notice de l'article provenant de la source Math-Net.Ru

Visual media possesses rich semantics and a remarkable ability to convey emotions and sentiments. This work, aims to recognize the sentiment conveyed by the visual content of an image. This task is quite complex as it requires extracting high-level abstract content from visual data. Both global and local regions within an image carry significant emotional cues. A saliency-based approach is utilized to predict visual attention, highlighting the most crucial local areas in an image. The proposed framework consists of two main modules: (1) feature extraction from global and local image regions using a pre-trained Darknet53 CNN, which captures high-level concepts, and (2) a two-stage feature selection process using a modified Sine Cosine Algorithm (SCA) and the ReliefF algorithm, followed by classification with an SVM classifier. The proposed Modified Sine Cosine Algorithm (MSCA) enhances the search path of the original SCA by introducing a new convergence empirical parameter, preventing the algorithm from getting trapped in local optima. The experimental findings, evaluated on four datasets, demonstrate that the bi-stage feature selection combined with a deep learning approach significantly improves the accuracy of sentiment classification.
Keywords: deep learning, Darknet53, sentiment analysis, local saliency information, modified SCA, ReliefF algorithm. 
@article{FSSC_2024_19_2_a0,
     author = {K. Usha Kingsly Devi and A. Mookambiga and J. Thirumal and V. Gomathi},
     title = {Improved sentiment classification of images using a two-stage relieff and modified {SCA} feature selection},
     journal = {Ne\v{c}etkie sistemy i m\^agkie vy\v{c}isleni\^a},
     pages = {53--87},
     publisher = {mathdoc},
     volume = {19},
     number = {2},
     year = {2024},
     language = {en},
     url = {http://geodesic.mathdoc.fr/item/FSSC_2024_19_2_a0/}
}
TY  - JOUR
AU  - K. Usha Kingsly Devi
AU  - A. Mookambiga
AU  - J. Thirumal
AU  - V. Gomathi
TI  - Improved sentiment classification of images using a two-stage relieff and modified SCA feature selection
JO  - Nečetkie sistemy i mâgkie vyčisleniâ
PY  - 2024
SP  - 53
EP  - 87
VL  - 19
IS  - 2
PB  - mathdoc
UR  - http://geodesic.mathdoc.fr/item/FSSC_2024_19_2_a0/
LA  - en
ID  - FSSC_2024_19_2_a0
ER  - 
%0 Journal Article
%A K. Usha Kingsly Devi
%A A. Mookambiga
%A J. Thirumal
%A V. Gomathi
%T Improved sentiment classification of images using a two-stage relieff and modified SCA feature selection
%J Nečetkie sistemy i mâgkie vyčisleniâ
%D 2024
%P 53-87
%V 19
%N 2
%I mathdoc
%U http://geodesic.mathdoc.fr/item/FSSC_2024_19_2_a0/
%G en
%F FSSC_2024_19_2_a0
K. Usha Kingsly Devi; A. Mookambiga; J. Thirumal; V. Gomathi. Improved sentiment classification of images using a two-stage relieff and modified SCA feature selection. Nečetkie sistemy i mâgkie vyčisleniâ, Tome 19 (2024) no. 2, pp. 53-87. http://geodesic.mathdoc.fr/item/FSSC_2024_19_2_a0/

[1] Zhao S., Gao Y., Ding G., Chua T. S., “Real-time multimedia social event detection in microblog”, IEEE transactions on cybernetics, 48:11 (2017), 3218–3231 | DOI

[2] Pour P. A., Hussain M. S., AlZoubi O., D'Mello S., Calvo R. A., “The impact of system feedback on learners' affective and physiological states”, International Conference on Intelligent Tutoring Systems, Springer, Berlin, Heidelberg, 2010, 264–273

[3] Klein J., Moon Y., Picard R. W., “This computer responds to user frustration: Theory, design, and results”, Interacting with computers, 14:2 (2002), 119–140 | DOI

[4] Morency L. P., Mihalcea R., Doshi P., “Towards multimodal sentiment analysis: Harvesting opinions from the web”, Proceedings of the 13th International Conference on Multimodal Interfaces, 2011, 169–176

[5] Yuan J., Mcdonough S., You Q., Luo J., “Sentribute: image sentiment analysis from a midlevel perspective”, Workshop on Issues of Sentiment Discovery and Opinion Mining, 2010, 10–10

[6] Mathews A. P., “Captioning Images Using Different Styles”, MM '15: Proceedings of the 23rd ACM international conference on Multimedia, 2015, 665–668 | DOI

[7] Sartori A., Culibrk D., Yan Y., Sebe N., “Who's afraid of Itten: Using the art theory of color combination to analyze emotions in abstract paintings”, MM '15: Proceedings of the 23rd ACM international conference on Multimedia, 2015, 311–320 | DOI

[8] Zhao S., Gao Y., Jiang X., Yao H., Chua T. -S., Sun X., “Exploring principles-of-art features for image emotion recognition”, MM '14: Proceedings of the 22nd ACM international conference on Multimedia, 2014, 47–56 | DOI

[9] Yanulevskaya V., van Gemert.J. C., Roth K., Herbold A. K., Sebe N., Geusebroek J. M., “Emotional valence categorization using holistic image features”, 2008 15th IEEE International Conference on Image Processing, 2008, 101–104 | DOI

[10] Machajdik J., Hanbury A., “Affective image classification using features inspired by psychology and art theory”, MM '10: Proceedings of the 18th ACM international conference on Multimedia, 2010, 83–92 | DOI

[11] Lu X., Suryanarayan P., Adams Jr.R. B., Li J., Newman M. G., Wang J. Z., “On shape and the computability of emotions”, MM '12: Proceedings of the 20th ACM international conference on Multimedia, 2012, 229–238 (in Russian) | DOI

[12] Chen T., Borth D., Darrell T., Chang S. F., Deepsentibank: Visual sentiment concept classification with deep convolutional neural networks, arXiv:1410.8586, 2014

[13] Krizhevsky A., Sutskever I., Hinton G. E., “ImageNet classification with deep convolutional neural networks”, Communications of the ACM, 60:6 (2017), 84–90 | DOI

[14] You Q., Luo J., Jin H., Yang J., “Joint visual-textual sentiment analysis with deep neural networks”, Proceedings of the 23rd ACM international conference on Multimedia, ACM, 2015, 1071–1074 | DOI

[15] You Q., Luo J., Jin H., Yang J., “Robust image sentiment analysis using progressively trained and domain transferred deep networks”, Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015

[16] Long J., Shelhamer E., Darrell T., “Fully convolutional networks for semantic segmentation”, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, 3431–3440

[17] Girshick R., Donahue J., Darrell T., Malik J., “Rich feature hierarchies for accurate object detection and semantic segmentation”, Proceedings of the IEEE conference on computer vision and pattern recognition, 2014, 580–587

[18] Cheng Z., Yang Q., Sheng B., “Deep colorization”, Proceedings of the IEEE International Conference on Computer Vision, 2015, 415–423

[19] Dong C., Loy C. C., He K., Tang X., “Learning a deep convolutional network for image super-resolution”, European Conference on Computer Vision, 2014, 184–199

[20] Zhou B., Lapedriza A., Xiao J., Torralba A., Oliva A., “Learning deep features for scene recognition using places database”, Advances in Neural Information Processing Systems, v. 27, eds. Z. Ghahramani, M. Welling, C. Cortes and N. Lawrence, K.Q. Weinberger, Curran Associates, Inc., 2014

[21] Campos V., Salvador A., Giro-i Nieto.X., Jou B., “Diving deep into sentiment: Understanding fine-tuned cnns for visual sentiment prediction”, Proceedings of the 1st International Workshop on Affect and Sentiment in Multimedia, 2015, 57–62 | DOI

[22] Peng K. -C., Chen T., Sadovnik A., Gallagher A. C., “A mixed bag of emotions: Model, predict, and transfer emotion distributions”, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, 860–868

[23] Xu C., Cetintas S., Lee K. -C., Li L. -J., Visual sentiment prediction with deep convolutional neural networks, arXiv: 1411.5731, 2014

[24] Joshi D., Datta R., Fedorovskaya E., Luong Q. -T., Wang J. Z., Li J., “Aesthetics and emotions in images”, IEEE Signal Processing Magazine, 28:5 (2011), 94–115 | DOI

[25] Li B., Xiong W., Hu W., Ding X., “Context-aware affective images classification based on bilayer sparse representation”, MM '12: Proceedings of the 20th ACM international conference on Multimedia, 2012, 721–724 | DOI

[26] Cambria E., Poria S., Gelbukh A., Thelwall M. A., “Sentiment analysis is a big suitcase”, IEEE Intelligent Systems, 32:6 (2017), 74–80 | DOI

[27] You Q., “Sentiment and emotion analysis for social multimedia: methodologies and applications”, Proceedings of the 2016 ACM on Multimedia Conference, 2016, 1445–1449

[28] Zheng H., Chen T., Luo J., When saliency meets sentiment: understanding how image content invokes emotion and sentiment, arXiv:1611.04636, 2016

[29] You Q., Jin H., Luo J., “Visual Sentiment Analysis by Attending on Local Image Regions”, Proceedings of the AAAI Conference on Artificial Intelligence, v. 33, 2017 | DOI

[30] Yang J., She D., Sun M., Cheng M. M., Rosin P., Wang L., “Visual Sentiment Prediction Based on Automatic Discovery of Affective Regions”, IEEE Transactions on Multimedia, 20:9 (2018), 2513–2525 | DOI

[31] Wu L., Qi M., Jian M., Zhang H., “Visual sentiment analysis by combining global and local information”, Neural Processing Letters, 51 (2020), 2063–2075 | DOI

[32] Shaojing F., Shen Z., Jiang M., Koenig B. L., Xu J., Kankanhalli M. S., Zhao Q., “Emotional attention: A study of image sentiment and visual attention”, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, 7521–7531 | DOI

[33] Song K., Yao T., Ling Q., Mei T., “Boosting image sentiment analysis with visual attention”, Neurocomputing, 312 (2018), 218–228 | DOI

[34] Jiang Z., Zaheer W., Wali A., Gilani S. A. M., “Visual sentiment analysis using data-augmented deep transfer learning techniques”, Multimedia Tools and Applications, 83 (2024), 17233–17249 | DOI

[35] Zhang H., Liu Y., Xiong Z., Wu Z., Xu D., “Visual sentiment analysis with semantic correlation enhancement”, Complex and Intelligent Systems, 10 (2024), 2869–2881 | DOI

[36] Fu K., Gu I. Y., Yang J., “Spectral salient object detection”, Neurocomputing, 275 (2018), 788–803 | DOI

[37] Kononenko I., “Estimating attributes: analysis and extensions of Relief”, Proceedings of European Conference on Machine Learning, Springer, Berlin, Heidelberg, 1994, 171–182

[38] Redmon J., Farhadi A., “Yolo9000: Better, faster, stronger”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, 6517–6525

[39] Mikels J. A., Fredrickson B. L., Larkin G. R., Lindberg C. M., Maglio S. J., Reuter-Lorenz P. A., “Emotional category data on images from the international affective picture system”, Behavior Research Methods, 37:4 (2005), 626–630 | DOI

[40] Mihalis N. A., Gunes H., Pantic M., “A multi-layer hybrid framework for dimensional emotion classification”, Proceedings of the 19th ACM international conference on Multimedia, 2011, 933–936

[41] He X., Zhang W., “Emotion recognition by assisted learning with convolutional neural networks”, Neurocomputing, 291 (2018), 187–194 | DOI

[42] Wang X., Jia J., Yin J., Cai L., “Interpretable aesthetic features for affective image classification”, IEEE International Conference on Image Processing, 2013, 3230–3234 | DOI

[43] Lu X., Suryanarayan P., Adams R. B., Li J., Newman M. G., Wang J. Z., “On shape and the computability of emotions”, MM '12: Proceedings of the 20th ACM international conference on Multimedia, 2012, 229–238 | DOI

[44] Borth D., Rongrong J., Chen T., Breuel T., Chang S. -F., “Largescale visual sentiment ontology and detectors using adjective noun pairs”, Proceedings of the 21st ACM International Conference on Multimedia, Association for Computing Machinery, New York, NY, USA, 2013, 223–232 | DOI

[45] Zhao S., Hongxun Y., You Y., Yanhao Z., “Affective image retrieval via multi-graph learning”, Proceedings of the 22nd ACM international conference on Multimedia, ACM, 2014, 1025–1028 | DOI

[46] Deng J., Dong W., Socher R., Li L. J., Li K., Fei-Fei L., “ImageNet: A large-scale hierarchical image database”, IEEE Conference on Computer Vision and Pattern Recognition, 2009, 248–255 | DOI

[47] Xu C., Cetintas S., Lee K. C., Li L. J., Visual sentiment prediction with deep convolutional neural networks, arXiv:1411.573137, 2014

[48] You Q., Jiebo L., Hailin J., Jianchao Y., “Robust image sentiment analysis using progressively trained and domain transferred deep networks”, Proceedings of the AAAI Conference on Artificial Intelligence, 29:1 (2015) | DOI

[49] Xiong H., Qing L., Shaoyi S., Yuanyuan C., “Region-based convolutional neural network using group sparse regularization for image sentiment classification”, EURASIP Journal on Image and Video Processing, 2019, no. 1, 1–9

[50] Jia Y., Shelhamer E., Donahue J., Karayev S., Long J., Girshick R., Guadarrama S., Darrell T., “Caffe: Convolutional architecture for fast feature embedding”, MM '14: Proceedings of the 22nd ACM international conference on Multimedia, 2014, 675–678 | DOI

[51] Rao T., Xiaoxu L., Min X., “Learning multi-level deep representations for image emotion classification”, Neural Processing Letters, 51 (2020), 2043–2061 | DOI

[52] Lu X., Lin Z., Jin H., Yang J., Wang J. Z., “Rating Image Aesthetics Using Deep Learning”, IEEE Transactions on Multimedia, 17:11 (2015), 2021–2034 | DOI

[53] Andrearczyk V., Whelan P. F., “Using filter banks in Convolutional Neural Networks for texture classification”, Pattern Recognition Letters, 84 (2016), 63–69 | DOI

[54] Fan S., Zhiqi S., Ming J., Bryan L. K., Juan X., Mohan S. K., Qi Z., “Emotional attention: A study of image sentiment and visual attention”, IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, 7521–7531 | DOI

[55] Song K., Ting Y., Qiang L., Tao M., “Boosting image sentiment analysis with visual attention”, Neurocomputing, 312 (2018), 218–228 | DOI

[56] Yang S., Xing L., Chang Z., Li Y., “Attention-Based Sentiment Region Importance and Relationship Analysis for Image Sentiment Recognition”, Computational Intelligence and Neuroscience, 2022, no. 1 | DOI

[57] Mirjalili S., “SCA: a sine cosine algorithm for solving optimization problems”, Knowledge-based systems, 96 (2016), 120–133 | DOI

[58] Li S., Huajing F., Xiaoyong L., “Parameter optimization of support vector regression based on sine cosine algorithm”, Expert systems with Applications, 91 (2018), 63–77 | DOI

[59] Hafez A. I., Hossam M. Z., Eid E., Aboul E. H., “Sine cosine optimization algorithm for feature selection”, 2016 international symposium on innovations in intelligent systems and applications (INISTA), IEEE, 2016, 1–5 (in Russian)

[60] Frintrop S., Rome E., Christensen H. I., “Computational visual attention systems and their cognitive foundations: A survey”, ACM Transactions on Applied Perception, 7:1 (2010), 6 | DOI

[61] Judd T., Ehinger K., Durand F., Torralba A., “Learning to predict where humans look”, IEEE 12th International Conference on Computer Vision, 2009, 2106–2113 | DOI

[62] Shi J., Malik J., “Normalized cuts and image segmentation”, IEEE Transactions on pattern analysis and machine intelligence, 22:8 (2000), 888–905 | DOI

[63] Redmon J., Divvala S., Girshick R., Farhadi A., “You only look once: Unified, real-time object detection”, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, 779–788

[64] He K., Zhang X., Ren S., Sun J., “Deep residual learning for image recognition”, IEEE Conference on Computer Vision and Pattern Recognition, 2016, 770–778

[65] Pan S. J., Shen D., Yang Q., Kwok J. T., “Transferring localization models across space”, Proceedings of the 23rd AAAI Conference on Artificial Intelligence, 2008, 1383–1388

[66] Belanche L. A., González-Navarro F. F., Review and Evaluation of Feature Selection Algorithms in Synthetic Problems, arXiv:1101.2320, 2011

[67] Kenji K., Rendell L. A., “The feature selection problem: Traditional methods and a new algorithm”, Proceedings of the AAAI Conference on Artificial Intelligence, 1992, 129–134

[68] Vapnik V., Isabel G., Trevor H., “Support vector machines”, Machine Learning, 20 (1995), 273–297 | DOI

[69] Lang P. J., Bradley M. M., Cuthbert B. N., International affective picture system (IAPS): Technical manual and affective ratings, University of Florida, Gainesville, 2008