Multimodal Depression Detection from WhatsApp Statuses Using Hybrid Feature Selection with HMO-RTH-OOA and KPCA-CCA
Abstract
Depression is a common and serious condition characterized by persistent sadness. Early detection is critical to leading a healthy and fulfilling life. Due to the rapid development of social media, more people are using it to share their thoughts and feelings. Therefore, social media sites like Facebook, Twitter, and Instagram are now developing into a significant data source that can be used for identifying depression and mental illness. In this work, we demonstrate the detection of depression using WhatsApp data. A dataset of 3 months' worth of images, text, and behavior of 100 users was collected with consent, labeled by three psychological professionals, and finalized by majority vote. We examined WhatsApp-based depression from two perspectives: (a) message-based, which uses a single WhatsApp status image with corresponding extracted text, and (b) user-based detection, which utilizes all status images, texts, and WhatsApp behavioral data of each user. BERT and ELECTRA models were employed for text feature extraction, while EfficientNetV2L was used for image feature extraction. We have proposed two feature selection approaches: a hybrid multi-objective Red-Tailed Hawk - Osprey Optimization Algorithm (HMORTH-OOA), which attempts to select an optimal set of features, and Kernel Principal Component Analysis - Canonical Correlation Analysis (KPCA-CCA), which reduces the dimensionality of the multi-modal dataset. Additionally, we employed data augmentation to improve the dataset count for message-based depression detection to 5514 samples. Classification was performed using standard machine learning models, including XGBoost (XGB), AdaBoost (ADA), Support Vector Machine (SVM), and Random Forest (RF). User-based depression detection achieved 93.79% accuracy with KPCA-CCA, while message-based depression detection achieved 90.46% accuracy with HMO-RTH-OOA, demonstrating significant improvements over depression detection performed on Twitter and Instagram datasets.
Full Text:
PDFReferences
Park, C.L., et al., Emotional well-being: What it is and why it matters. Affective Science, 2023. 4(1): p. 10-20, doi: 10.1007/s42761-022-00163-0
Katchapakirin, K., et al. Facebook social media for depression detection in the Thai community. in 2018 15th international joint conference on computer science and software engineering (jcsse). 2018. IEEE, doi: 10.1109/JCSSE.2018.8457362
Tronick, E.Z., Emotions and emotional communication in infants. Parent-infant psychodynamics, 2018: p. 35-53, doi: 10.1037//0003-066X.44.2.112
Keles, B., N. McCrae, and A. Grealish, A systematic review: the influence of social media on depression, anxiety and psychological distress in adolescents. International journal of adolescence and youth, 2020. 25(1): p. 79-93, doi: 10.1080/02673843.2019.1590851
Elbay, R.Y., et al., Depression, anxiety, stress levels of physicians and associated factors in Covid-19 pandemics. Psychiatry research, 2020. 290: p. 113130, doi: 10.1016/j.psychres.2020.113130
Ríssola, E.A., D.E. Losada, and F. Crestani, A survey of computational methods for online mental state assessment on social media. ACM Transactions on Computing for Healthcare, 2021. 2(2): p. 1-31, doi: https://doi.org/10.1145/3437259
Zogan, H., et al. Depressionnet: learning multi-modalities with user post summarization for depression detection on social media. in proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval. 2021, doi: 10.1145/3404835.3462938
Kim, J., et al., A systematic review of the validity of screening depression through Facebook, Twitter, Instagram, and Snapchat. Journal of Affective Disorders, 2021. 286: p. 360-369, doi: 10.1016/j.jad.2020.08.091
Takkar, H., et al. MemoChat App: An Advancement in Existing WhatsApp. in 2024 1st International Conference on Advances in Computing, Communication and Networking (ICAC2N). 2024. IEEE, doi: 10.1109/ICAC2N63387.2024.10895869
Susandri, S., S. Defit, and M. Tajuddin, Enhancing Text Sentiment Classification with Hybrid CNN-BiLSTM Model on WhatsApp Group. Journal of Advances in Information Technology, 2024. 15(3) , doi: 10.12720/jait.15.3.355-363
Saeidi, S., Identifying personality traits of WhatsApp users based on frequently used emojis using deep learning. Multimedia Tools and Applications, 2024. 83(5): p. 13873-13886, doi: 10.1007/s11042-023-15209-z
Ay, B., et al., Automated depression detection using deep representation and sequence learning with EEG signals. Journal of medical systems, 2019. 43: p. 1-12, doi: 10.1007/s10916-019-1345-y
Wu, P., et al., Automatic depression recognition by intelligent speech signal processing: A systematic survey. CAAI Transactions on Intelligence Technology, 2023. 8(3): p. 701-711, doi:10.1049/cit2.12113
Wang, Q., H. Yang, and Y. Yu, Facial expression video analysis for depression detection in Chinese patients. Journal of Visual Communication and Image Representation, 2018. 57: p. 228-233, , doi: doi.org/10.1016/j.jvcir.2018.11.003
Babu, N.V. and E.G.M. Kanaga, Sentiment analysis in social media data for depression detection using artificial intelligence: a review. SN Computer Science, 2022. 3: p. 1-20, doi: 10.1007/s42979-021-00958-1
Feng, C., et al., Pre-trained language embedding-based contextual summary and multi-scale transmission network for aspect extraction. Procedia Computer Science, 2020. 174: p. 40-49 , doi: 10.1016/j.procs.2020.06.054
Reece, A.G. and C.M. Danforth, Instagram photos reveal predictive markers of depression. EPJ Data Science, 2017. 6(1): p. 15. , doi: 10.1140/epjds/s13688-017-0110-z
Lin, C., et al. Sensemood: depression detection on social media. in Proceedings of the 2020 international conference on multimedia retrieval. 2020, doi: 10.1145/3372278.3391932
Omipidan, I.A. and B.O. Sanusi, Rise of social media in the digital age: Whatsapp a threat to effective communication. IMSU Journal of Communication Studies, 2024. 8(1): p. 142-153, doi: 10.5281/zenodo.12693727
Ferahtia, S., et al., Red-tailed hawk algorithm for numerical optimization and real-world problems. Scientific Reports, 2023. 13(1): p. 12950, doi: 10.1038/s41598-023-38778-3
Dehghani, M. and P. Trojovský, Osprey optimization algorithm: A new bio-inspired metaheuristic algorithm for solving engineering optimization problems. Frontiers in Mechanical Engineering, 2023. 8: p. 1126450, doi: 10.3389/fmech.2022.1126450
Cortiz, D., Exploring transformers in emotion recognition: a comparison of bert, distillbert, roberta, xlnet and electra. arXiv preprint arXiv:2104.02041, 2021, doi: 10.48550/arXiv.2104.02041
Zhang, S., H. Yu, and G. Zhu, An emotional classification method of Chinese short comment text based on ELECTRA. Connection Science, 2022. 34(1): p. 254-273, doi: 10.1080/09540091.2021.1985968
Zhou, Y., et al., Developing a machine learning model for detecting depression, anxiety, and apathy in older adults with mild cognitive impairment using speech and facial expressions: A cross-sectional observational study. International journal of nursing studies, 2023. 146: p. 104562, doi: 10.1016/j.ijnurstu.2023.104562
Abbas, Q., et al., Brain and Heart Rate Variability Patterns Recognition for Depression Classification of Mental Health Disorder. International Journal of Advanced Computer Science & Applications, 2024. 15(7) , doi: 10.14569/IJACSA.2024.0150784
Yang, J., et al., Depression detection based on analysis of EEG signals in multi brain regions. Journal of Integrative Neuroscience, 2023. 22(4): p. 93, doi: 10.31083/j.jin2204093
Tasci, G., et al., Automated accurate detection of depression using twin Pascal’s triangles lattice pattern with EEG Signals. Knowledge-Based Systems, 2023. 260: p. 110190, doi: 10.1016/j.knosys.2022.110190
Espinola, C.W., et al., Detection of major depressive disorder using vocal acoustic analysis and machine learning—an exploratory study. Research on Biomedical Engineering, 2021. 37: p. 53-64, doi: 10.1007/s42600-020-00100-9
Xu, X., et al., Leveraging collaborative-filtering for personalized behavior modeling: a case study of depression detection among college students. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2021. 5(1): p. 1-27, doi: 10.1145/3448107
William, D. and D. Suhartono, Text-based depression detection on social media posts: A systematic literature review. Procedia Computer Science, 2021. 179: p. 582-589, doi: 10.1016/j.procs.2021.01.043
Zhang, Z., et al., Multimodal sensing for depression risk detection: Integrating audio, video, and text data. Sensors, 2024. 24(12): p. 3714. , doi: 10.3390/s24123714
Ricard, B.J., et al., Exploring the utility of community-generated social media content for detecting depression: an analytical study on Instagram. Journal of medical Internet research, 2018. 20(12): p. e11817. doi:10.2196/11817
Lup, K., L. Trub, and L. Rosenthal, Instagram# instasad?: exploring associations among instagram use, depressive symptoms, negative social comparison, and strangers followed. Cyberpsychology, Behavior, and Social Networking, 2015. 18(5): p. 247-252, doi: 10.1089/cyber.2014.0560
Sasso, M.P., et al., # Sad: Twitter content predicts changes in cognitive vulnerability and depressive symptoms. Cognitive Therapy and Research, 2019. 43: p. 657-665, doi: 10.1007/s10608-019-10001-6
Smith, R.J., et al., Variations in Facebook posting patterns across validated patient health conditions: a prospective cohort study. Journal of medical Internet research, 2017. 19(1): p. e7, doi: 10.2196/jmir.6486
Ding, Y., et al., A depression recognition method for college students using deep integrated support vector algorithm. IEEE access, 2020. 8: p. 75616-75629 , doi: 10.1109/ACCESS.2020.2987523
Skaik, R. and D. Inkpen. Using twitter social media for depression detection in the canadian population. in Proceedings of the 2020 3rd Artificial Intelligence and Cloud Computing Conference. 2020, doi: 10.1145/3442536.3442553
Liaw, A.S. and H.N. Chua. Depression detection on social media with user network and engagement features using machine learning methods. in 2022 IEEE International Conference on Artificial Intelligence in Engineering and Technology (IICAIET). 2022. IEEE, doi: 10.1109/IICAIET55139.2022.9936814
Adarsh, V., et al., Fair and explainable depression detection in social media. Information Processing & Management, 2023. 60(1): p. 103168, doi: doi.org/10.1016/j.ipm.2022.103168
Maxim, S., N. Ignatiev, and I. Smirnov. Predicting depression with social media images. in Proc. 9th Int. Conf. Pattern Recognit. Appl. Methods. 2020, doi: 10.5220/0009168602350240
Gui, T., et al. Cooperative multimodal approach to depression detection in twitter. in Proceedings of the AAAI conference on artificial intelligence. 2019, doi: 10.1609/aaai.v33i01.330111
An, M., et al. Multimodal topic-enriched auxiliary learning for depression detection. in proceedings of the 28th international conference on computational linguistics. 2020, doi: 10.18653/v1/2020.coling-main.94
Huang, Y.-C., C.-F. Chiang, and A.L. Chen, Predicting Depression Tendency based on Image, Text and Behavior Data from Instagram. DATA, 2019. 3(1): p. 32-40, doi: 10.5220/0007833600320040
De Choudhury, M., et al. Predicting depression via social media. in Proceedings of the international AAAI conference on web and social media. 2013, doi: 10.1609/icwsm.v7i1.14432
Kang, K., C. Yoon, and E.Y. Kim. Identifying depressive users in Twitter using multimodal analysis. in 2016 international conference on big data and smart computing (BigComp). 2016. IEEE, doi: 10.1109/BIGCOMP.2016.7425918
Safa, R., P. Bayat, and L. Moghtader, Automatic detection of depression symptoms in twitter using multimodal analysis. The Journal of Supercomputing, 2022. 78(4): p. 4709-4744, doi: 10.1109/BIGCOMP.2016.7425918
Mamidisetti, S. and A.M. Reddy, Enhancing Depression Prediction Accuracy Using Filter and Wrapper-Based Visual Feature Extraction. Journal of Advances in Information Technology, 2023. 14(6), doi: 10.12720/jait.14.6.1425-1435
Wah, Y.B., et al., Feature selection methods: Case of filter and wrapper approaches for maximising classification accuracy. Pertanika Journal of Science & Technology, 2018. 26(1).
Arora, S., et al., A new hybrid algorithm based on grey wolf optimization and crow search algorithm for unconstrained function optimization and feature selection. Ieee Access, 2019. 7: p. 26343-26361, doi: 10.1109/ACCESS.2019.2897325
Gaidhane, P.J. and M.J. Nigam, A hybrid grey wolf optimizer and artificial bee colony algorithm for enhancing the performance of complex systems. Journal of computational science, 2018. 27: p. 284-302, doi: doi.org/10.1016/j.jocs.2018.06.008
Comert, S.E. and H.R. Yazgan, A new approach based on hybrid ant colony optimization-artificial bee colony algorithm for multi-objective electric vehicle routing problems. Engineering Applications of Artificial Intelligence, 2023. 123: p. 106375, doi: 10.1016/j.engappai.2023.106375
Dogan, G. and F.P. Akbulut, Multi-modal fusion learning through biosignal, audio, and visual content for detection of mental stress. Neural Computing and Applications, 2023. 35(34): p. 24435-24454, doi: 10.1007/s00521-023-09036-4
Safavipour, M.H., M.A. Doostari, and H. Sadjedi, A hybrid approach to multimodal biometric recognition based on feature-level fusion of face, two irises, and both thumbprints. Journal of Medical Signals & Sensors, 2022. 12(3): p. 177-191, doi: 10.4103/jmss.jmss_103_21
Meng, H., et al., Facial expression recognition algorithm based on fusion of transformed multilevel features and improved weighted voting SVM. Mathematical Problems in Engineering, 2021. 2021(1): p. 6639598, doi: 10.1155/2021/6639598
Zhang, X., et al., Fusing of electroencephalogram and eye movement with group sparse canonical correlation analysis for anxiety detection. IEEE Transactions on Affective Computing, 2020.
(2): p. 958-971, doi: 10.1109/TAFFC.2020.2981440
Devi, M.S., et al. Efficient Net Transfer Learning based Early Prediction of Monkey Pox Lesion. in 2022 3rd International Conference on Smart Electronics and Communication (ICOSEC). 2022. IEEE, doi: 10.1109/ICOSEC54921.2022.9952099
Pricolo, P., et al., Whole-body magnetic resonance imaging (WB-MRI) reporting with the METastasis Reporting and Data System for Prostate Cancer (MET-RADS-P): inter-observer agreement between readers of different expertise levels. Cancer Imaging, 2020. 20: p. 1-13, doi: 10.1186/s40644-020-00350-x
Shen, J. and F. Xu, Method of fault feature selection and fusion based on poll mode and optimized weighted KPCA for bearings. Measurement, 2022. 194: p. 110950, doi: 10.1016/j.measurement.2022.110950
Altaf, M., et al., A new statistical features based approach for bearing fault diagnosis using vibration signals. Sensors, 2022. 22(5): p. 2012, doi: 10.3390/s22052012
DOI: https://doi.org/10.31449/inf.v46i18.9236
This work is licensed under a Creative Commons Attribution 3.0 License.