Robotics: Science and Systems XXI

LangWBC: Language-directed Humanoid Whole-Body Control via End-to-end Learning

Yiyang Shao, Bike Zhang, Qiayuan Liao, Xiaoyu Huang, Yuman Gao, Yufeng Chi, Zhongyu Li, Sophia Shao, Koushil Sreenath

Abstract:

General-purpose humanoid robots are expected to interact intuitively with humans, enabling seamless integration into daily life. Natural language provides the most accessible medium for this purpose. However, translating language into humanoid whole-body motion remains a significant challenge, primarily due to the gap between linguistic understanding and physical actions. In this work, we present an end-to-end, language-directed policy for real-world humanoid whole-body control. Our approach combines reinforcement learning with policy distillation, allowing a single neural network to interpret language commands and execute corresponding physical actions directly. To enhance motion diversity and compositionality, we incorporate a Conditional Variational Autoencoder (CVAE) structure. The resulting policy achieves agile and versatile wholebody behaviors conditioned on language inputs, with smooth transitions between various motions, enabling adaptation to linguistic variations and the emergence of novel motions. We validate the efficacy and generalizability of our method through extensive simulations and real-world experiments, demonstrating robust whole-body control. Please see our website at LangWBC.github.io for more information.

Download:

Bibtex:

  
@INPROCEEDINGS{ShaoY-RSS-25, 
    AUTHOR    = {Yiyang Shao AND Bike Zhang AND Qiayuan Liao AND Xiaoyu Huang AND Yuman Gao AND Yufeng Chi AND Zhongyu Li AND Sophia Shao AND Koushil Sreenath}, 
    TITLE     = {{LangWBC: Language-directed Humanoid Whole-Body Control via End-to-end Learning}}, 
    BOOKTITLE = {Proceedings of Robotics: Science and Systems}, 
    YEAR      = {2025}, 
    ADDRESS   = {LosAngeles, CA, USA}, 
    MONTH     = {June}, 
    DOI       = {10.15607/RSS.2025.XXI.065} 
}