Research Highlight:

A tale of two senses—a look at the confluence and malleability of multisensory stimuli

Original Research Article:
AR Powers III, AR Hillock and MT Wallace (2009). Perceptual training narrows the temporal window of multisensory binding. J Neurosci. 29 (39): 12265-12274.

In our daily activities, whether it is driving down the road, playing a sport or enjoying a meal at our favorite restaurant, the brain is constantly inundated with multisensory information that it shrewdly processes to comprehend every sensory experience. Just how information from the multiple senses is integrated in the brain to produce a cohesive perceptual experience is intriguing. In particular, the temporal dynamics of the individual sensory information has to uniquely fall within a certain binding window in order for the brain to perceive it as a stimulus from the same environmental event. A better understanding of how binding of cross-modal cues takes place in relation to this temporal window is imperative, especially in the clinical realm where an enlargement of this window has been reported for various neurobiological disorders like dyslexia and autism. Research scientists at Vanderbilt University recently investigated the temporal characteristic and malleability of this multisensory binding window by using an auditory and visual stimulus pair in two perceptual training paradigms.

Powers et al. first tested normal adults on a 2-alternative forced choice (2-AFC) paradigm where participants had to make a simultaneity judgment about whether the occurrence of a visual and auditory stimulus was “simultaneous” or “non-simultaneous”. This assessment phase was followed by a simultaneity judgment training phase where the participants were given feedback about whether their judgment was correct. A subset of participants was requested to return one week later for another follow up assessment phase. The authors were successfully able to use this 2-AFC paradigm to define and characterize a multisensory temporal binding window profile. Moreover, they found that the training received during the 2-AFC task actually narrowed this multisensory binding window suggesting the flexibility of neural processes to adapt to feedback during training. The follow up assessment conducted one week after the training resulted in the maintenance of this temporal binding window suggesting the stability of brain processes to maintain a learned perceptual experience. However, the modifiable nature of this temporal binding window was not observed for passive exposure to identical stimuli that did not require a simultaneity judgment with feedback.

While these results showed an interesting and unique finding, one question that still remained unanswered was whether the flexible nature of this temporal binding window was a result of inherent cognitive biases. To further explore the nature of this multisensory temporal binding window and rule out the possibility that participants had a cognitive bias towards one particular multisensory task, Powers and colleagues ran a second experiment, a two-interval forced choice(2-IFC) perceptual paradigm. This paradigm was similar to the 2-AFC in that they used exactly the same stimuli, however, the participants were presented with two visual-auditory pairs, one that was simultaneously presented and the other that was not simultaneously presented. Participants were instructed to indicate which of the two presentations contained the simultaneous multisensory stimulus. This assessment session was then followed by a training phase where the participants were given feedback as to the accuracy of their responses after each trial. A subset of these participants also returned one week later to do a similar simultaneity judgment assessment.

The authors found similar results to those seen in the 2-AFC. The perceptual training in the 2-IFC training task resulted in a significant narrowing of the multisensory temporal binding window. Moreover, these modified binding widow changes seemed to remain stable even after the one week follow-up assessment. Powers and colleagues keenly observed that there was no significant difference in the degree and time course of window narrowing across the two different paradigms suggesting that the neural mechanisms underlying the malleability of this binding window is similar across different experimental contingencies.

So what does this tell us about human multisensory perceptual experiences? This study has successfully shown that perceptual training paradigms can create lasting changes in a person’s judgment of perceived simultaneity between visual and auditory events. More notably, the perceptual discrimination abilities prompted by training is not a result of simple exposure to passive stimuli, but the result of feedback on the accuracy of the simultaneity judgment. Seminal studies have shown significant reorganization of cortical space when a constrained set of stimuli were used early in development. However, a passive exposure to the same set of stimuli did not produce any notable behavioral change or neural reorganization. Powers and colleagues found significant effects in the temporal dynamics of the multisensory binding window after only one day of training, with stable effects observed even after one week. This alludes to the fact that the brain is capable of adapting quickly to new environmental situations and long term memory consolidation may play a role in strengthening learned traits. Thus results from this study would support the idea that the pairing of a sensory stimulus with behavioral salience (like feedback in this study) is crucial for sensory reorganization of adult cortical space.

It is intriguing to consider where and how the plasticity associated with this temporal binding window is modulated. In humans, recent neuroimaging studies reported a large, dynamic network of areas including the insular cortex, posterior parietal and superior temporal cortices, all critically involved in the perception of audiovisual stimuli. The neuronal oscillations among different cortical populations have also been shown to play a potential role in multisensory processing and temporal binding. At a more cellular level, the temporal tuning profile of multisensory neurons has been associated with adult plasticity in various sensory systems, with basal cholinergic signals acting as an instructive cue. This indicates that the synchronous role of both cortical and subcortical mechanisms may be responsible for temporal plasticity in multisensory systems.

The present work by Powers and colleagues and future extensions of similar studies holds particular clinical significance, especially in designing tailored intervention strategies for disorders such as dyslexia, autism and schizophrenia where altered multisensory temporal processes have been observed.