Zero-shot style transfer for gesture animation driven by text and speech using adversarial disentanglement of multimodal style encoding.

ABSTRACT: Modeling virtual agents with behavior style is one factor for personalizing human-agent interaction. We propose an efficient yet effective machine learning approach to synthesize gestures driven by prosodic features and text in the style of different speakers including those unseen during training. Our model performs zero-shot multimodal style transfer driven by multimodal data from the PATS database containing videos of various speakers. We view style as being pervasive; while speaking, it colors the communicative behaviors expressivity while speech content is carried by multimodal signals and text. This disentanglement scheme of content and style allows us to directly infer the style embedding even of a speaker whose data are not part of the training phase, without requiring any further

SUBMITTER: Fares M 

PROVIDER: S-EPMC10291316 | biostudies-literature | 2023

REPOSITORIES: biostudies-literature