Robotics: Science and Systems XXII

ELVIS: Ensemble-Calibrated Latent Imagination for Long-Horizon Visual MPC

Yurui Du, Pinhao Song, Yutong Hu, Renaud Detry

Abstract:

A central challenge of visual control with model-based reinforcement learning (RL) is reliable long-horizon planning: Long rollouts with learned latent dynamics exhibit branching futures and multi-modal action-value distributions. In addition, compounding model error amplified by visual occlusions make deep imagination brittle. We present ELVIS, a latent model predictive controller (MPC) designed to make long-horizon planning practical. ELVIS plans in a Dreamer-style recurrent state space model (RSSM) and replaces standard unimodal model predictive path integral (MPPI) with a Gaussian-mixture MPPI that maintains multiple coherent hypotheses over long horizons, avoiding mode averaging under branching rollouts. In parallel, ELVIS stabilizes deep imagination with a shared uncertainty-aware λ_t-return: an ensemble of latent critics defines an upper-confidence-bound (UCB) score that gates a time-varying λ_t, adaptively trading off bootstrapping versus look-ahead to limit compounding error during planning. The same return is used both to train an actor-critic prior from imagined rollouts and to score candidate trajectories inside GMM-MPPI, aligning RL objectives with the planner’s long-horizon optimization. On fourteen DeepMind Control Suite visual tasks, ELVIS establishes state-of-the-art performance compared with TD-MPC2 and DreamerV3. Finally, ELVIS transfers zero-shot to a real-world sand spraying task with severe occlusions, improving surface-quality metrics and demonstrating robustness beyond simulation.

Download:

Bibtex:

  
@INPROCEEDINGS{DuY-RSS-26, 
    AUTHOR    = {Yurui Du AND Pinhao Song AND Yutong Hu AND Renaud Detry}, 
    TITLE     = {{ELVIS: Ensemble-Calibrated Latent Imagination for Long-Horizon Visual MPC}}, 
    BOOKTITLE = {Proceedings of Robotics: Science and Systems}, 
    YEAR      = {2026}, 
    ADDRESS   = {Sydney, Australia}, 
    MONTH     = {July}, 
    DOI       = {10.15607/RSS.2026.XXII.182} 
}