Cross-Modal Sentiment Analysis on Social Media using Improved Nonverbal Representation Learning and GHRNN Fusion

Abstract

Traditional sentiment analysis methods mainly focus on textual data, while human emotions are multidimensional, usually related to sound, body language, etc. The use of multi-modal data can provide a deeper and broader understanding of human sentiment conveyance. To address the above issues, a method for extracting and analyzing emotional information features based on improved nonverbal representation learning networks and multi-modal data (Improved Nonverbal Representation Learning Networks and MD, INPNRLN-MD) is proposed. On this basis, an improved multi-modal data fusion method based on Gated Hierarchical Recurrent Neural Network and Cross-Modal Attention (GHRNN-CMA) is designed for the MD fusion part. Compared with traditional baselines, INPNRLN-MD extracts text features through the BERT model and utilizes ELN to process audio and video data, which can more effectively capture emotional information in multi-modal data. The cross-modal attention mechanism of GHRNN-CMA can enhance the interaction between modalities and improve the accuracy of emotion information recognition. Finally, the performance of the model is validated on the CMU-OSI and CMU-MCSEI datasets using indicators such as F1 value, Pearson correlation, mean absolute error, and second-order accuracy. During the training process, the study used a single NVIDIA GTX TITAN X GPU for testing, with 12 VRAM and a batch size of only 32 to converge. The inference stage has a relatively light computational load and can be deployed on ordinary cloud servers or edge devices. The research results show that compared with mainstream algorithms, the emotion information feature extraction and analysis method based on improved nonverbal representation learning network and multi-modal data performs the best, with F1 score, Pearson correlation, average absolute error, and second-order accuracy reaching 83.06/85.12, 0.803, 0.696, and 83.17/85.23, respectively. The average absolute error, Pearson correlation, F1 score, and second-order accuracy of the improved multi-modal data fusion method have been improved by 1.0%, 14.67%, 3.1%, and 3.3% respectively compared to the latest method. The above results indicate that research methods are helpful in perceiving and analyzing human emotions, which is beneficial for understanding and predicting human behavior in the future, and is of great significance for maintaining social relationships and improving social governance.