Robotics: Science and Systems X

Open-vocabulary Object Retrieval

Sergio Guadarrama, Erik Rodner, Kate Saenko, Ning Zhang, Ryan Farrell, Jeff Donahue, Trevor Darrell


In this paper, we address the problem of retrieving objects based on open-vocabulary natural language queries: Given a phrase describing a specific object, e.g., ``the corn flakes box'', the task is to find the best match in a set of images containing candidate objects. When naming objects, humans tend to use natural language with rich semantics, including basic-level categories, fine-grained categories, and instance-level concepts such as brand names. Existing approaches to large-scale object recognition fail in this scenario, as they expect queries that map directly to a fixed set of pre-trained visual categories, e.g. ImageNet synset tags. We address this limitation by introducing a novel object retrieval method. Given a candidate object image, we first map it to a set of words that are likely to describe it, using several learned image-to-text projections. We also propose a method for handling open-vocabularies, i.e., words not contained in the training data. We then compare the natural language query to the sets of words predicted for each candidate and select the best match. Our method can combine category- and instance-level semantics in a common representation. We present extensive experimental results on several datasets using both instance-level and category-level matching and show that our approach can accurately retrieve objects based on extremely varied open-vocabulary queries. The source code of our approach will be publicly available together with pre-trained models at \url{} and could be directly used for robotics applications.



    AUTHOR    = {Sergio Guadarrama AND Erik Rodner AND Kate Saenko AND Ning Zhang AND Ryan Farrell AND Jeff Donahue AND Trevor Darrell}, 
    TITLE     = {Open-vocabulary Object Retrieval}, 
    BOOKTITLE = {Proceedings of Robotics: Science and Systems}, 
    YEAR      = {2014}, 
    ADDRESS   = {Berkeley, USA}, 
    MONTH     = {July},
    DOI       = {10.15607/RSS.2014.X.041}