RSS 2009

Marc Ernst

The Puzzle of Human Multisensory Perception: Optimal Integration for Action
Marc Ernst


We use all our senses to construct a reliable percept representing the world with which we interact. The view we take in the Independent Max-Planck Research Group for Human Multisensory Perception and Action is that in many aspects of behaviour, motor actions and multisensory processing are inseparably linked and therefore have to be studied in a closed action/perception loop. We believe that human perception and action is tailored to the statistics of the natural environment and when the environment changes our perceptions will follow these changes through the process of adaptation minimizing potential costs during interaction. In the neural processing such statistics will represent itself in probability distributions. We follow Hermann von Helmholtz in our belief that human perception is a problem of inference, for which the sensory data are often not sufficient to uniquely determine the percept. Thus, prior knowledge has to be used to constrain the process of inference from ambiguous sensory signals. A principled way to describe the combination of prior knowledge with sensory data in a probabilistic way is the Bayesian Framework. Therefore, we regularly use this Bayesian Framework to construct "ideal observer" models– models that use the available information in the most optimal way, provided some task and cost function. These models can then be used as a benchmark against which human performance can be tested. To do so in the Multisensory Perception and Action Group we use quantitative psychophysical and neuropsychological methods together with Virtual Reality techniques. Quantitative psychophysical methods are important to best determine the relevant perceptual parameters minimizing uncertainty and unknowns. Virtual Reality is important because it provides us with a tool to precisely control the perceptual situation that are investigated, while at the same time it allows for a degree of interaction, which is necessary for studying the action/perception loop. Often, however, today's Virtual Reality techniques and Human-Computer Interaction devices are not sufficiently developed to be readily used in the study of human perception and action. Therefore, some of our work concentrates on the development of human-machine interfaces. In this talk I will provide some overview of how humans use multisensory information in an optimal way in order to guide actions.


Marc Ernst studied Physics in Heidelberg and Frankfurt/Main. In 2000 he received his Ph.D. degree at the Max Planck Institute for Biological Cybernetics for investigations on human visuomotor behavior. For this work he was awarded the Attempto-Prize (2000) from the University of Tübingen and the Otto-Hahn-Medaille (2001) from the Max Planck Society. Starting in 2000, he spent almost 2 years as a postdoc at the University of California, Berkeley working with Prof. Martin Banks on psychophysical experiments and computational models investigating the integration of visual-haptic information. End of 2001, Marc Ernst returned to the Max Planck Institute and became principle investigator of the Sensorimotor Lab in the Department of Prof. Heinrich Bülthoff. Beginning 2007 he then became leader of the Independent Max Planck Research Group on Human Multisenory Perception and Action. The group consists of is five Postdocs, three PhD, and two master students, and is interested in human multimodal perception, sensorimotor integration and men-machine interaction. The group participates in several international collaborative grants, including the EU Project ImmerSence, investigating human-human and human-machine interaction, and a HFSP project focusing on perceptual learning. Furthermore, Marc Ernst was coordinating the 6^th Framework IST European Project "CyberWalk", which developed an omnidirectional treadmill in order to enable natural free walking through Virtual Environments. Marc Ernst is Vice President and founding member of the Eurohaptics Society and Vice Chair of the IEEE Technical Committee on Haptics. He is member of IEEE, the Robotics and Automation Society, and the Vision Science Society. Furthermore, Marc Ernst is member of the Advisory Council of the International Association for the Study of Attention and Performance and of the newly founded excellence Center for Integrative Neuroscience in Tübingen.