Robotics: Science and Systems XXI

Demonstrating GPU Parallelized Robot Simulation and Rendering for Generalizable Embodied AI with ManiSkill3

Stone Tao, Fanbo Xiang, Arth Shukla, Yuzhe Qin, Xander Hinrichsen, Xiaodi Yuan, Chen Bao, Xinsong Lin, Yulin Liu, Tse-Kai Chan, Yuan Gao, Xuanlin Li, Tongzhou Mu, Nan Xiao, Arnav Gurha, Viswesh Nagaswamy Rajesh, Yong Woo Choi, Yen-Ru Chen, Zhiao Huang, Roberto Calandra, Rui Chen, Shan Luo, Hao Su

Abstract:

Simulation has enabled unprecedented computescalable approaches to robot learning. However, many existing simulation frameworks typically support a narrow range of scenes/tasks and lack features critical for scaling generalizable robotics and sim2real. We introduce and open source ManiSkill3, the fastest state-visual GPU parallelized robotics simulator with contact-rich physics targeting generalizable manipulation. ManiSkill3 supports GPU parallelization of many aspects including simulation+rendering, heterogeneous simulation, pointclouds/voxels visual input, and more. GPU Simulation with rendering on ManiSkill3 uses 2-3x less GPU memory usage than other platforms and achieves up to 30,000+ FPS in benchmarked environments due to minimal python/pytorch overhead in the system, simulation on the GPU, and the use of the SAPIEN parallel rendering system. Tasks that used to take hours to train can now take minutes. We further provide the most comprehensive range of GPU parallelized environments/tasks spanning 12 distinct domains including but not limited to mobile manipulation, drawing, humanoids, and dextrous manipulation in realistic scenes designed by artists or real-world digital twins. In addition, millions of demonstration frames are provided from motion planning, RL, and teleoperation. ManiSkill3 also provides a comprehensive set of baselines that span popular RL and learning-from-demonstrations algorithms.

Download:

Bibtex:

  
@INPROCEEDINGS{TaoS-RSS-25, 
    AUTHOR    = {Stone Tao AND Fanbo Xiang AND Arth Shukla AND Yuzhe Qin AND Xander Hinrichsen AND Xiaodi Yuan AND Chen Bao AND Xinsong Lin AND Yulin Liu AND Tse-Kai Chan AND Yuan Gao AND Xuanlin Li AND Tongzhou Mu AND Nan Xiao AND Arnav Gurha AND Viswesh Nagaswamy Rajesh AND Yong Woo Choi AND Yen-Ru Chen AND Zhiao Huang AND Roberto Calandra AND Rui Chen AND Shan Luo AND Hao Su}, 
    TITLE     = {{Demonstrating GPU Parallelized Robot Simulation and Rendering for Generalizable Embodied AI with ManiSkill3}}, 
    BOOKTITLE = {Proceedings of Robotics: Science and Systems}, 
    YEAR      = {2025}, 
    ADDRESS   = {LosAngeles, CA, USA}, 
    MONTH     = {June}, 
    DOI       = {10.15607/RSS.2025.XXI.021} 
}