Authors: Mansi, Assistant Professor Dr. Grace M. BasanShrieh
Abstract: No single source of information captures the full experience of depression or anxiety. Artificial intelligence (AI) research has therefore turned to multiple digital signals, including standardised questionnaires, electronic health records, written language, speech, facial behaviour and passively sensed smartphone and wearable data. This paper compares these modalities for AI-assisted detection of depression and anxiety. Drawing on the literature synthesised in a doctoral study, it maps each modality to the symptom domains it can reflect (cognitive, emotional, behavioural, physiological and functional), summarises the features typically extracted, and appraises each against six practical criteria: clinical validity, scalability, participant burden, privacy protection, interpretability and cross-cultural robustness. It also compares early, late and hybrid fusion strategies. The analysis shows that the modalities are complementary rather than interchangeable. Questionnaires and language are strongest for cognitive and emotional symptoms, speech and facial signals for psychomotor and affective expression, and wearables for behavioural and physiological rhythms. Late fusion of modality-specific models offers the best balance of robustness to missing data and auditability for early clinical research. The paper concludes with design recommendations for building multimodal datasets that can support valid, fair and clinically interpretable AI screening.
