Head nods in spoken and signed interaction using CV

In this project we analyze head nods in German Sign Language (DGS), Russian Sign Language (RSL) and in spoken German and Russian using Computer Vision. We identify that head nods simultaneously occur with various lexical and non-lexical manual forms, but more frequently head nods are produced on their own or replace the manual signs. The aim of the study is two-fold. First, we describe the basic phonetic properties of head nods alongside their interaction with manual signs. Secondly, we investigate whether the phonetic properties of head nods fulfilling various functions differ significantly. We hypothesize that head nods signaling affirmation are to have a larger amplitude than head nods functioning as feedback in interaction. We define feedback very broadly as interactional moves that display some kind of stance towards the information represented by another signer. In our data we thus consider feedback signals in different functions (e.g., as continuers, acknowledgement tokens, change-of-state tokens, assessments and repairs). Head nods signaling feedback are assumed to have a smaller amplitude but a longer duration of the movement and to be produced without co-occurring manual signs.
To test this prediction, we use the body pose information. Based on this information, which was automatically generated using the Computer Vision (CV) tool OpenPose (Cao et al., 2019), we calculate head nods measurements from video recordings and investigate the head nods in terms of the number of peaks, the amplitude of the nod, the frequency and the duration of these head movements. We also inspect the spectral qualities of head nods through the use of Short-time Fourier transforms. Prior to the analysis, we identified all head nods in the data and annotated them manually in ELAN. Consequently, we used the pose information to compute statistics about head nodding to further analyze phonetic properties of these movements quantitatively.
Our preliminary results show that different phonological types of head nods exist in DGS. After testing the applicability of OpenPose for the analysis of head nods in DGS, our next steps will include applying the same method to analyzing more manual and non-manual elements in our data from spoken language interaction to further complement the ongoing work of the GeSi project distinguishing similar constructions in sign and spoken language.
Collaborations with: Anna Kuder, Marc Schulder, Job Schepens
Latest outcome
Bauer, Anastasia, Anna Kuder, Marc Schulder & Job Schepens (in prep). Exploring the Influence of Communicative Function on Nonmanual Gesture’s Form: insights from the study on head nods in spoken German and DGS.
Bauer, Anastasia, Anna Kuder, Marc Schulder & Job Schepens. 2024. Phonetic differences between affirmative and feedback head nods in German Sign Language (DGS): A pose estimation study. PLOS ONE 19(5): e0304040. https://doi.org/10.1371/journal.pone.0304040
Bauer, Anastasia, Kuder, Anna, Schulder, Marc, & Job Schepens. 2024. Supplementary materials for "Phonetic differences between affirmative and feedback head nods in German Sign Language (DGS): A pose estimation study" (1.0) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.10838848