Autor:innen:
Inka Camilla Hiß, Aachen (Germany)
Christian Pilz, Aachen (Germany)
Ulrich Canzler, Aachen (Germany)
Jarek Krajewski, Düsseldorf (Germany)
Andre Wittenborn, Düsseldorf (Germany)
Klaus Steffen Leonhardt, Aachen (Germany)
Christoph Weiss, Aachen (Germany)
Ute Habel, Aachen (Germany)
Benjamin Clemens, Aachen (Germany)
Patients with different forms of psychopathology often display similar impairments in the
expression and regulation of emotions. For example, affective flattening, anhedonia and
apathy are prevalent symptoms in major depression, bipolar disorder, schizophrenia and
schizoaffective disorder. Current diagnostic methods for these psychopathologies mostly rely
on observed behaviour and clinical interviews. This approach, however, fails to incorporate
relevant physiological and neuro-biological parameters, which could provide diagnostically
relevant, biologically grounded information on affective states.
The aim of this study is to develop a diagnostic framework that actively combines detailed
assessments of relevant clinical factors, sociodemographic information, audio-visual and
physiological parameters. We hypothesize, that this multimodal diagnostic framework allows
for a more personalized and sensitive diagnostic process. Multivariate pattern classification
was used to jointly analyse specific visual (head movement dynamics, facial movement
dynamics), auditory (prosody of speech) and physiological (skin perfusion and heart rate
variability (HRV)) parameters of patients diagnosed with either depression (n=47) or
schizophrenia (n=33). At the beginning and end of their inpatient stay, patients were recorded
in realistic conditions, i.e. during a standardized clinical interview (Hamilton Rating Scale for
Depression). The full recording set-up, including multiple RGB cameras, a near-infrared
camera, finger pulse oximeter and a high-quality microphone, can be seen in Figure 1.
First results indicate that a super-vector combining facial movement dynamics, prosody of
speech and HRV is most strongly correlated to symptom severity in both, depression and
schizophrenia. These findings suggest that the combination of these three parameters might
provide a potential transdiagnostic biomarker for affective states, allowing us to differentiate
between depression and schizophrenia based on a combination of auditory, visual and
physiological parameters. Compared to the standard diagnostic process, such a multimodal
approach allows for a more precise diagnosis and provides a first step towards personalized,
transdiagnostic markers for affective states.