An effect of multisensory training on visual processing (Commentary on Grasso et al. (2016))

Animals use their multiple senses in parallel to maximize the quantity and variety of information gathered from the environment. When an event is registered by more than one sense, information about it can be synthesized across the senses to derive more accurate estimates of its features (e.g. its location and identity), facilitating swifter and more appropriate responses to it. Given its functional utility, it is unsurprising that multisensory processing is not only ubiquitous across species, but is found throughout the brain at multiple levels of the neuraxis.

Much research has been devoted to determining how, when, and where the outputs of unisensory systems are integrated within multisensory circuits (Stein, 2012). There is an inherent challenge in coordinating signals transduced from different energies and processed by unisensory circuits with different operational parameters (e.g. time constants). At least part of the solution to this challenge is that internal circuits use experience with cross-modal cues to adapt their internal configurations (Stein et al., 2014). This sensitivity to multisensory experience is most potent in the neonate, but persists throughout life. Yet, the end goal of these computations is not just the formation of multisensory representations to direct decisions and overt responses. In parallel with these efforts to understand how unisensory signals are engaged by multisensory processing, there has been a growing appreciation that the output of multisensory circuits can, in turn, enhance unisensory processing and representations in a number of ways (Schroeder & Foxe, 2005; Shams et al., 2011; Thelen et al., 2015).

The present author group previously combined these ideas in a spectacular series of experiments in which they rehabilitated patients suffering large unilateral lesions of visual cortex, lesions which rendered them functionally blind to visual stimuli in contralateral space (hemianopia) (see review in Dundon et al., 2015). Amazingly, non-invasive multisensory training (involving repeated exposures of spatiotemporally concordant visual-auditory cues) can restore some basic visual processing capabilities within the previously blind hemifield. A pathway identified as a likely contributor was one originating from a multisensory structure, the midbrain superior colliculus (SC), and terminating within the classically defined ‘dorsal stream’ of visual processing. If this projection was involved in the rehabilitation, a natural prediction is that multisensory training should also be capable of selectively altering visual processing within the dorsal stream in neurotypic subjects.

In this issue of EJN, Grasso et al. (2016) examine the changes that multisensory training induces in event-related potentials derived from areas associated with the dorsal stream while subjects perform a motion discrimination task and, as a control, areas associated with the ‘ventral stream’ while subjects perform an orientation discrimination task. During multisensory training, subjects were repeatedly exposed to visual and auditory cues presented alone or simultaneously in spatially concordant or (for different subjects) discordant visual-auditory pairs. Visual-auditory pairs were asymmetrically distributed to be more frequent on a ‘trained’ vs. ‘untrained’ side of space (3 : 1 ratio). Their main finding is, after training, a significant increase in the amplitude of the N1 component over dorsal stream areas during the motion discrimination task, but only in the ‘trained’ hemisphere, and only when training involved concordant visual-auditory cues. The N1 component never significantly changed when training involved disparate cues, or in the ‘untrained’ hemisphere, or in the orientation discrimination task. This demonstrates that physiological responses within the dorsal stream are sensitive to concordant multisensory training in neurotypic subjects.

This modulation of unisensory processing by multisensory training might take place entirely within the cortex, or it might be mediated by a colliculo-cortical pathway. Additional support for this link comes from the study of individual SC neurons subjected to a similar multisensory training paradigm: repeatedly exposing visual-auditory SC neurons to concordant visual-auditory cues can enhance their responsiveness to the individual visual and auditory components (Yu et al., 2013). One could speculate that concordant multisensory training could enhance the unisensory visual responses of the SC which could, via some route, confer enhanced processing capabilities to the dorsal stream in the cortex during the motion discrimination task. How this signal might get from the multisensory (i.e. intermediate, deep) layers of the SC to the cortex is unknown, although several possibilities exist (Krauzlis et al., 2013).

There are at least two additional interesting features of this experiment. The first is that the training paradigm did not contain explicit feedback or reward, suggesting the operation of an unsupervised learning algorithm. Still, it is possible that this training might be altered or augmented by higher order processes. The second feature is that the ‘untrained’ side in the experiment also received some multisensory training (25% of the visual-auditory trials), it was just insufficient to produce the main effect. Why? One possibility is that the raw number of visual-auditory exposures on the untrained side failed to cross some threshold. An alternative is that plasticity within each hemisphere is sensitive to the local probabilities of stimulus co-occurrence. When visual and auditory cues are presented individually and independently with equal frequency, there is a 50% probability that an auditory cue randomly appears when a visual cue appears. Based on the uneven inter-hemispheric distribution of visual-auditory pairs during training, this probability increased to 60% (above chance) in the trained hemisphere and decreased to 33% in the untrained hemisphere (below chance). This might be another possibility for future exploration, as the probability of co-occurrence is a powerful indicator of whether the cues are related (Xu et al., 2012; Kayser & Shams, 2015). Indeed, in studies of SC physiology, exclusively cross-modal exposure (100% probability of co-occurrence) has been the most robust multisensory training paradigm (Yu et al., 2009, 2013).

In closing, Grasso et al. (2016) provide a nice, well-controlled demonstration that training with paired visual-auditory cues can change the effectiveness with which the brain processes other visual cues in a completely different task paradigm. This alone is an intriguing result. However, these results are particularly interesting in the context of the group's other work, and are consistent with the idea that the SC might play a role in modulating activity within the dorsal stream of visual processing.