Objective Metrics for Evalutating Gesture Generation are Almost Useless

How do you evaluate something as subjective and ephemeral as human body language for natural and lifelike qualities?

There are many approaches to generating gestures for game characters based on speech. But the lack of of a standard way of judging the appropriateness and human-like qualities of those gestures has been a stumbling block to progress in the field.

The GENEA Challenge on speech-driven gesture generation brings together researchers who use different methods to generate and evaluate non-verbal behaviour. The goal is to stimulate discussion about how to improve both the generation methods and the evaluation of the results.

SEED’s Taras Kucherenko has been instrumental in creating and advancing the GENEA Challenge and talks about some recent surprising results when comparing automated tools with user studies.

Research Paper

The research paper on the GENEA Challenge was published in ACM Transactions on Graphics, April 2024. Authors: Taras Kucherenko, Pieter Wolfert, Youngwoo Yoon, Carla Viegas, Teodor Nikolov, Mihail Tsakov, and Gustav Eje Henter.

Check out the GENEA Challenge 2022 paper here.