Robotics: Science and Systems XVII

MQA: Answering the Question via Robotic Manipulation

Yuhong Deng^*, Di Guo^*, Xiaofeng Guo, Naifu Zhang, Huaping Liu, Fuchun Sun
^* These authors contributed equally

Abstract:

In this paper; we propose a novel task; Manipulation Question Answering (MQA); where the robot performs manipulation actions to change the environment in order to answer a given question. To solve this problem; a framework consisting of a QA module and a manipulation module is proposed. For the QA module; we adopt the method for the Visual Question Answering (VQA) task. For the manipulation module; a Deep Q Network (DQN) model is designed to generate manipulation actions for the robot to interact with the environment. We consider the situation where the robot continuously manipulating objects inside a bin until the answer to the question is found. Besides; a novel dataset that contains a variety of object models; scenarios and corresponding question-answer pairs is established in a simulation environment. Extensive experiments have been conducted to validate the effectiveness of the proposed framework.

Download:

Bibtex:

  
@INPROCEEDINGS{DengGuo-RSS-21, 
    AUTHOR    = {Yuhong Deng AND Di Guo AND Xiaofeng Guo AND Naifu Zhang AND Huaping Liu AND Fuchun Sun}, 
    TITLE     = {{MQA: Answering the Question via Robotic Manipulation}}, 
    BOOKTITLE = {Proceedings of Robotics: Science and Systems}, 
    YEAR      = {2021}, 
    ADDRESS   = {Virtual}, 
    MONTH     = {July}, 
    DOI       = {10.15607/RSS.2021.XVII.044} 
}