EPV562
Large Language Model in Automatic Depression Detection
S. H. Ling 1,*, W. Chorney 2
1School of Medicine, University of Limerick, Limerick, 2College of Medicine and Health, University College Cork , Cork, Ireland
Introduction: Depression affects over 280 million people globally and is linked to substantial morbidity, mortality, and poor outcomes in comorbid conditions. Despite the availability of validated screening tools, up to 63.6% of cases remain undiagnosed, delaying timely treatment. Automated depression screening using text or voice data has emerged to improve accessibility and detection. Large language models (LLMs) have shown promise in identifying depression from social media, clinical notes, and structured interviews. However, current performance estimates are most likely optimistic due to relying on commercial models.
Objectives: This study evaluates the feasibility and accuracy of using privacy-preserving locally deployed LLMs with realistic size constraints for automated depression detection using clinical dialogue transcripts.
Methods: We employed the Distress Analysis Interview Corpus–Wizard-of-Oz (DAIC-WOZ) dataset containing therapist–patient transcripts labelled for depression. Several open-source LLMs under 15 billion parameters (Gemma3, Qwen3, DeepSeek-R1, Phi3, and LLaMA3.2) were deployed locally via Ollama. Each model received a standardized system prompt to classify each transcript as “depressed” (1) or “not depressed” (0). Predictions were repeated ten times per model. Accuracy, sensitivity, specificity, precision, F1 score, and Cohen’s kappa were calculated with 95% confidence intervals using Scikit-learn and Scipy (α = 0.05).
Results: Each model is trained and validated on the Distress Analysis Interview Corpus/Wizard of Oz dataset using locally deployed LLMs to preserve privacy. Best performing models achieved 67.7% accuracy, 46.2% precision, 72.9% specificity (deepseek-r1:14b), 92.9% sensitivity (gemma3:1b), and 56.4% F1 (gemma3:12b).
Image:
Image 2:
Image 3:
Conclusions: The results demonstrated that current local LLMs are not adequate for clinical utility in automatic depression screening. Future studies are required to fine tune locally operated LLMs for depression screening.
Disclosure of Interest: None Declared