Robotics: Science and Systems XXII

Steerable Vision-Language-Action Policies for Embodied Reasoning and Hierarchical Control

William Chen, Jagdeep Singh Bhatia, Catherine Glossop, Nikhil Mathihalli, Andy Tang, Ria Doshi, Danny Driess, Karl Pertsch, Sergey Levine

Abstract:

Pretrained vision-language models (VLMs) can make semantic and visual inferences across diverse settings, providing valuable common-sense priors for robotic control. However, effectively grounding this knowledge in robot behaviors remains an open challenge. Prior methods often employ a hierarchical approach where VLMs reason over high-level commands to be executed by separate low-level policies, e.g., vision-language-action models (VLAs). The interface between VLMs and VLAs is usually *natural language task instructions*, which fundamentally limits how much VLM reasoning can steer low-level behavior. We thus introduce Steerable Policies: VLAs trained on rich synthetic commands at various levels of abstraction, like subtasks, motions, and grounded pixel coordinates. By improving low-level controllability, Steerable Policies can unlock pretrained knowledge in VLMs, enabling improved task generalization. We demonstrate this benefit by controlling our Steerable Policies with both a learned high-level embodied reasoner and an off-the-shelf VLM prompted to reason over command abstractions via in-context learning. Across extensive real-world manipulation experiments, these two novel methods outperform prior embodied reasoning VLAs and VLM-based hierarchical baselines, including on challenging generalization and long-horizon tasks.

Download:

Bibtex:

  
@INPROCEEDINGS{ChenW-RSS-26, 
    AUTHOR    = {William Chen AND Jagdeep Singh Bhatia AND Catherine Glossop AND Nikhil Mathihalli AND Andy Tang AND Ria Doshi AND Danny Driess AND Karl Pertsch AND Sergey Levine}, 
    TITLE     = {{Steerable Vision-Language-Action Policies for Embodied Reasoning and Hierarchical Control}}, 
    BOOKTITLE = {Proceedings of Robotics: Science and Systems}, 
    YEAR      = {2026}, 
    ADDRESS   = {Sydney, Australia}, 
    MONTH     = {July}, 
    DOI       = {10.15607/RSS.2026.XXII.074} 
}