Robotics: Science and Systems XXII

Simulation Distillation: Pretraining World Models in Simulation for Rapid Real-World Adaptation

Jacob Levy, Tyler Westenbroek, Kevin Huang, Fernando Palafox, Patrick Yin, Shayegan Omidshafiei, Dong-Ki Kim, Abhishek Gupta, David Fridovich-Keil

Abstract:

Simulation-to-real transfer remains a central challenge in robotics, as mismatches between simulated and real-world dynamics often lead to failures. While reinforcement learning offers a principled mechanism for adaptation, existing sim-to-real finetuning methods struggle with exploration and long-horizon credit assignment in the low-data regimes typical of real-world robotics. We introduce \texttt{Simulation Distillation} (\texttt{SimDist}), a sim-to-real framework that distills structural priors from a simulator into a latent world model and enables rapid real-world adaptation via online planning and supervised dynamics finetuning. By transferring reward and value models directly from simulation, \texttt{SimDist} provides dense planning signals from raw perception without requiring value learning during deployment. As a result, real-world adaptation reduces to short-horizon system identification, avoiding long-horizon credit assignment and enabling fast, stable improvement. Across precise manipulation and quadruped locomotion tasks, \texttt{SimDist} substantially outperforms prior methods in data efficiency, stability, and final performance.

Download:

Bibtex:

  
@INPROCEEDINGS{LevyJ-RSS-26, 
    AUTHOR    = {Jacob Levy AND Tyler Westenbroek AND Kevin Huang AND Fernando Palafox AND Patrick Yin AND Shayegan Omidshafiei AND Dong-Ki Kim AND Abhishek Gupta AND David Fridovich-Keil}, 
    TITLE     = {{Simulation Distillation: Pretraining World Models in Simulation for Rapid Real-World Adaptation}}, 
    BOOKTITLE = {Proceedings of Robotics: Science and Systems}, 
    YEAR      = {2026}, 
    ADDRESS   = {Sydney, Australia}, 
    MONTH     = {July}, 
    DOI       = {10.15607/RSS.2026.XXII.017} 
}