Alignment of speech and co-speech gesture in a constraint-based grammar
This thesis concerns the form-meaning mapping of multimodal communicative actions consisting of speech signals and improvised co-speech gestures, produced spontaneously with the hand. The interaction between speech and speech-accompanying gestures has been standardly addressed from a cognitive perspective to establish the underlying cognitive mechanisms for the synchronous speech and gesture production, and also from a computational perspective to build computer systems that communicate through multiple modalities. Based on the findings of this previous research, we advance a new theory in which the mapping from the form of the combined speech-and-gesture signal to its meaning is analysed in a constraint-based multimodal grammar. We propose several construction rules about multimodal well-formedness that we motivate empirically from an extensive and detailed corpus study. In particular, the construction rules use the prosody, syntax and semantics of speech, the form and meaning of the gesture signal, as well as the temporal performance of the speech relative to the temporal performance of the gesture to constrain the derivation of a single multimodal syntax tree which in turn determines a meaning representation
Saint-Amand, Katya
speech ; gesture ; multimodal ; grammar
Alignment of speech and co-speech gesture in a constraint-based grammar. This thesis concerns the form-meaning mapping of multimodal communicative actions consisting of speech signals and improvised co-speech gestures, produced spontaneously with the hand. The interaction between speech and speech-accompanying gestures has been standardly addressed from a cognitive perspective to establish the underlying cognitive mechanisms for the synchronous speech and gesture production, and also from a computational perspective to build computer systems that communicate through multiple modalities. Based on the findings of this previous research, we advance a new theory in which the mapping from the form of the combined speech-and-gesture signal to its meaning is analysed in a constraint-based multimodal grammar. We propose several construction rules about multimodal well-formedness that we motivate empirically from an extensive and detailed corpus study. In particular, the construction rules use the prosody, syntax and semantics of speech, the form and meaning of the gesture signal, as well as the temporal performance of the speech relative to the temporal performance of the gesture to constrain the derivation of a single multimodal syntax tree which in turn determines a meaning representation
alignment speech speech gesture constraint_based grammar concern form_meaning mapping multimodal communicative_action consist speech_signal speech gesture produce hand interaction speech speech gesture standardly address cognitive_perspective establish underlying cognitive_mechanism synchronous speech gesture production also computational perspective build computer system communicate multiple modality based finding previous advance new theory mapping form combine speech and gesture signal meaning analyse constraint_based multimodal grammar propose several construction rule multimodal well formedness motivate empirically extensive detail corpus_study particular construction rule use prosody syntax_semantics speech form_meaning gesture signal well temporal performance speech relative temporal performance gesture constrain derivation single multimodal syntax tree turn determines mean representation