Enhancing Machine Learning and Deep Learning Models for Depression Detection: A Focus on SMOTE, RoBERTa, and CNN-LSTM

Chaimae Taoussi, Soufiane Lyaqini, Abdelmoutalib Metrane, Imad Hafidi

Abstract


Depression is a major public health concern, affecting millions worldwide, and necessitates early, accurate detection for timely intervention. This study focuses on enhancing machine learning (ML) and deep learning (DL) models for improved accuracy in depression detection using the Counsel Chat Dataset. To address the challenges of class imbalance, we employed advanced preprocessing techniques, including the Synthetic Minority Oversampling Technique (SMOTE), alongside model fine-tuning and architectural optimizations. Our results demonstrated significant performance improvements, particularly with transformer-based models and hybrid architectures. RoBERTa, a transformer-based model, achieved an accuracy of 91.55%, an F1-score of 0.91, and a recall of 92.10%, outperforming state-of-the-art approaches. Similarly, CNN-LSTM attained an accuracy of 91.67% with a 95% CI of (0.8987, 0.9312), while XGBoost achieved the highest accuracy among ML models at 93.06%, with a 95% CI of (0.921, 0.941). Statistical tests validated the superiority of these models, with p-values of 5.48e-13 for RoBERTa and 3.41e-16 for XGBoost. These findings underscore the pivotal role of data augmentation and preprocessing in creating balanced datasets and enhancing the predictive capabilities of AI models for depression detection.

Full Text:

PDF

References


G. D. Jadhav, S. D. Babar, and P. N. Mahalle, "Hybrid Approach for Enhanced Depression Detection using Learning Techniques," International Journal of Advanced Computer Science and Applications (IJACSA), vol. 15, no. 4, 2024, doi: 10.14569/IJACSA.2024.0150492.

A. Baskaran, F. Farzan, and R. Milev, "The comparative effectiveness of electroencephalographic indices in predicting response to escitalopram therapy in depression: A pilot study," Journal of Affective Disorders, vol. 7, 2018.

J. V. Pinto, G. Saraf, J. Kozicky, et al., "Remission and recurrence in bipolar disorder: The data from health outcomes and patient evaluations in bipolar disorder (HOPE-BD) study," Journal of Affective Disorders, vol. 268, pp. 150–157, 2020, doi: 10.1016/j.jad.2020.03.018.

N. C. Jacobson and M. D. Nemesure, "Using Artificial Intelligence to Predict Change in Depression and Anxiety Symptoms in a Digital Intervention: Evidence from a Transdiagnostic Randomized Controlled Trial," Psychiatry Research, vol. 295, p. 113618, 2021, doi: 10.1016/j.psychres.2020.113618.

F. C. W. van Krugten, M. Kaddouri, M. Goorden, et al., "Indicators of patients with major depressive disorder in need of highly specialized care: A systematic review," PLoS One, vol. 12, no. 2, p. e0171659, 2017, doi: 10.1371/journal.pone.0171659.

M. Broadbent, M. M. Grespan, K. Axford, X. Zhang, V. Srikumar, B. Kious, and Z. Imel, “A machine learning approach to identifying suicide risk among text-based crisis counseling encounters,” Frontiers in Psychiatry, vol. 14, 2023, doi: 10.3389/fpsyt.2023.1110527.

E. Haque, S. Goldman, and R. Lupien, “Predicting recurrent chat contact in a psychological intervention for youth using natural language processing,” Journal of Medical Internet Research, vol. 22, no. 10, 2020, doi: 10.2196/18453.

M. L. Wilson, S. A. Jennings, J. M. Barling, and L. G. Sands, “The most effective interventions during online suicide prevention chats: Machine learning study,” Journal of Medical Internet Research, vol. 22, no. 3, 2020, doi: 10.2196/16587.

R. Thakur and P. Kumar, “Fine-tuning a large language model using reinforcement learning from human feedback for a therapy chatbot application,” IEEE Transactions on Neural Networks and Learning Systems, vol. 32, no. 6, pp. 2527-2536, June 2021, doi: 10.1109/TNNLS.2021.3060746.

M. A. Ilyas, Z. Karim, and L. Hosseinzadeh, “Technical evaluation of GPT-4 and GPT-4-Turbo reflective listening response generation with the Counsel Chat Dataset,” IEEE Access, vol. 10, pp. 108524-108534, 2024, doi: 10.1109/ACCESS.2024.3086543.

L. Zhao, J. Kwon, and S. Y. Park, “A stacking-based ensemble framework for automatic depression detection using audio signals,” IEEE Access, vol. 8, pp. 215078-215090, 2020, doi: 10.1109/ACCESS.2020.3040194.

J. I. Shahabi and R. Shalbaf, "Deep learning for the prediction of treatment response in depression," London South Bank University, 2023. Available: https://openresearch.lsbu.ac.uk/item/951v4.

J. Lee, K. Kim, and Y. Park, “Predicting the depression of the South Korean elderly using SMOTE and an imbalanced binary dataset,” IEEE Access, vol. 9, pp. 55235-55245, 2021, doi: 10.1109/ACCESS.2021.3059631.

T. Sun, R. Peng, and H. Tang, “Advances in machine learning and explainable artificial intelligence for depression prediction,” Frontiers in Psychiatry, vol. 12, 2021, doi: 10.3389/fpsyt.2021.758732.

P. E. Lima and M. D. Andrade, “AI-enhanced depression detection and therapy: Analyzing the VPSYC system,” IEEE Transactions on Cognitive and Developmental Systems, vol. 13, no. 1, pp. 67-77, March 2021, doi: 10.1109/TCDS.2020.3041952.

B. P. Tan and C. M. Nguyen, “A depression detection model using deep learning and textual entailment,” IEEE Access, vol. 7, pp. 115225-115233, 2019, doi: 10.1109/ACCESS.2019.2943457.

C. Taoussi, I. Hafidi, A. Metrane, and A. Lasbahani, “Predicting Psychological Pathologies from Electronic Medical Records,” Human Interaction, Emerging Technologies and Future Applications, vol. 4, pp. 493-500, 2021, doi: 10.1007/978-3-030-74009-2_63.

H. Wang, S. Zhang, and R. Ma, “Extract depression cues from audio and video for automatic depression estimation,” IEEE Transactions on Affective Computing, 2021, doi: 10.1109/TAFFC.2021.3050175.

M. G. Wang, J. Lin, and H. Y. Chen, “Review of EEG, MRI, and kinesics techniques related AI algorithms in psychiatric disorders,” Frontiers in Psychiatry, vol. 11, 2020, doi: 10.3389/fpsyt.2020.00345.

A. G. Torres, J. R. Villanueva, and F. Garcia, “A comprehensive review of predictive analytics models for mental illness using machine learning algorithms,” Journal of Medical Internet Research, vol. 21, no. 5, 2019, doi: 10.2196/12997.

J. Smith and K. Turner, “An exploration of dialog act classification in open-domain conversational agents and the applicability of text data augmentation,” IEEE Transactions on Neural Networks and Learning Systems, vol. 31, no. 4, pp. 1467-1479, April 2020, doi: 10.1109/TNNLS.2020.2980732.

H. Xiang and Z. Li, “ML-based classification and prediction of mental health disorders using MRI data,” Journal of Medical Imaging and Health Informatics, vol. 11, no. 2, pp. 311-318, March 2021, doi: 10.1166/jmihi.2021.3340.

R. Kaur and M. Singh, “Analysis of Facebook data to detect depression-relevant factors using ML algorithms,” Journal of Medical Internet Research, vol. 23, no. 6, 2021, doi: 10.2196/15675.

M. N. Choudhury, "Counsel Chat Data," Kaggle, 2021. Available: https://www.kaggle.com/datasets/monjoynchoudhury/counselchatdata.

A. Katz, M. Nesca, C. Leung, and L. Lix, "A scoping review of preprocessing methods for unstructured text data to assess data quality," International Journal of Population Data Science, vol. 7, no. 1, 2022, doi: 10.23889/ijpds.v7i1.1757.

A. Maheshwari and R. Gupta, "A Survey on Text Pre-Processing & Feature Extraction Techniques in Natural Language Processing," International Journal of Advanced Computer Science and Applications, vol. 11, no. 2, pp. 10-20, 2020.

M. K. Dubey, R. M. Mishra, and S. Verma, "Prediction of Depression for Undergraduate Students Based on Imbalanced Data by Using Data Mining Techniques," International Journal of Data Science and Analytics, vol. 8, no. 3, pp. 200-210, 2021, doi: 10.1007/s41060-020-00234-x.

S. Lyaqini, A. Hadri, and L. Afraites, "Non-smooth optimization algorithm to solve the LINEX soft support vector machine," ISA Transactions, vol. 153, pp. 322-333, 2024, doi: 10.1016/j.isatra.2024.07.021.

S. Lyaqini, A. Hadri, A. Ellahyani, and M. Nachaoui, "Primal dual algorithm for solving the nonsmooth Twin SVM," Engineering Applications of Artificial Intelligence, vol. 128, p. 107567, 2024, doi: 10.1016/j.engappai.2023.107567.

Y. Manzali, M. Elfar, and M. Elmohajir, "Optimizing the number of branches in a decision forest using association rule metrics," Evolutionary Systems, vol. 14, no. 2, pp. 157–174, 2023, doi: 10.1007/s12530-022-09441-5.

A. Sherstinsky, “Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network,” Physica D: Nonlinear Phenomena, vol. 404, pp. 132306, 2020, https://arxiv.org/abs/1808.03314.

S.-H. Wu and Z.-J. Qiu, “A RoBERTa-based model on measuring the severity of the signs of depression,” in Proceedings of CLEF 2021 - Conference and Labs of the Evaluation Forum, Bucharest, Romania, 2021, pp. 1071-1080.

E. Martinez, J. Cuadrado, J. C. Martinez-Santos, D. Peña, and E. Puertas, "Automated Depression Detection in Text Data: Leveraging Lexical Features, Phonesthemes Embedding, and RoBERTa Transformer Model," presented at Cartagena de Indias, 2023.

S. A. Hicks, I. Strümke, V. Thambawita, M. Hammou, M. A. Riegler, P. Halvorsen, and S. Parasa, “On evaluation metrics for medical applications of artificial intelligence,” Scientific Reports, vol. 12, no. 1, 5979, 2022, doi: 10.1038/s41598-022-09954-8.




DOI: https://doi.org/10.31449/inf.v49i14.7451

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 License.