Journal of Neurophysiology

Touching Sounds: Thalamocortical Plasticity and the Neural Basis of Multisensory Integration

Marcus J. Naumer, Jasper J. F. van den Bosch


To date, noninvasive neuroimaging research on multisensory perception has focused on cortical activations. In a series of elegant functional magnetic resonance imaging experiments, Beauchamp and Ro recently investigated altered cortical activations associated with acquired sound–touch synesthesia resulting from a thalamic lesion. Their findings highlight the important role of intact thalamocortical projections for preventing illusory crossmodal perception and for underlying reliable multisensory integration.

Our senses provide us with detailed information about objects and events in our environment. The effective integration of information across sensory-modality boundaries is of vital importance. The basic principles of multisensory integration (e.g., the spatiotemporal proximity of multiple inputs) have been revealed in many single-cell studies and specifically in studies of the cat superior colliculus (for a review see Stein and Stanford 2008). During the last decade, noninvasive neuroimaging has given investigators the ability to study the neural basis of multisensory integration in humans. These studies have provided converging evidence that multisensory interactions occur even at the lowest levels of the cortical processing hierarchy (for a review see Driver and Noesselt 2008). However, there are surprisingly few studies testing the role of subcortical structures (e.g., the cerebellum or thalamus) in multisensory integration.

In a recent article, Beauchamp and Ro (2008) reported on a patient (SR) with a rare infarct restricted to the ventrolateral nucleus of her right thalamus (Fig. 1). As one would have expected, this infarct had initially resulted in a loss of somatosensory sensation on the contralateral half of SR's body. Fortunately, this deficit almost completely disappeared within a period of about 18 mo. However, concurrent with this improvement in tactile sensation, the patient developed symptoms of auditory–tactile synesthesia, where certain sounds induced intense and often unpleasant somatosensory tingling sensations in her left hand and arm. Auditory–tactile synesthesia is one of the rarest forms of synesthesia.

The authors hypothesized that crossmodal plasticity resulted in inappropriate structural links developing between these sensory modalities. This assumption is supported by Beauchamp and Ro's earlier diffusion tensor imaging (DTI) measurements in SR (Ro et al. 2007), which had already revealed a disruption of white-matter tracts between the lesioned right somatosensory thalamus and somatosensory cortical regions.

In the current study, patient SR (now 6 yr after the stroke) and a group of nine control subjects participated in a series of three experiments. In the first, subjects listened to a variety of natural and artificial sounds, including animal vocalizations, man-made object sounds, scrambled sounds, and pure tones. Patient SR showed significantly enhanced functional magnetic resonance imaging (fMRI) signal amplitudes (compared with the mean activation in normal controls) in the parietal operculum, a secondary somatosensory (SII) region. Earlier studies using invasive electrophysiology in nonhuman primates (Schroeder et al. 2001) and fMRI in humans (e.g., Foxe et al. 2002) had already implicated similar opercular regions in processes of normal auditory–tactile integration. However, further evidence for the multisensory integrative capacities of SII came from a study of Keysers and colleagues (2004) who had demonstrated that this region could also be activated visually by solely using gray-scale images of humans touching others. Moreover, Beauchamp and Ro revealed that auditory activation differences between patient SR and the controls were most pronounced in distinct opercular subregions (OP1 and OP4) of SII (see Fig. 1, G and H in Beauchamp and Ro 2008).

To investigate the effects in greater detail, the authors conducted a second auditory experiment. Here, Beauchamp and Ro made use of visual choice displays that allowed SR to report the intensity and location of her tactile sensations. To avoid motor contaminations of the fMRI signal, visual fixations (recorded using an eye-tracking system) were used instead of conventional button presses. Most interestingly, this experiment revealed an almost linear relationship between the strength of SR's subjective tactile sensations and the fMRI signal amplitudes in the opercular subregions OP1 and OP4.

Finally, the authors conducted a control experiment where they attached piezoelectric vibrotactile stimulators to the hands and feet of the patient and each of their control subjects. These stimulators produced mild sensations comparable to holding a vibrating cell phone, but without any accompanying sound. This time, the subjects had to report the intensity and location of the vibrotactile stimuli. Like the control subjects, patient SR reported strong vibrations at each stimulated location, with a detection rate of 100%. However, in the patient's SII region far fewer voxels were activated compared with the average activation in the control group. The largest differential effect occurred in OP1. Differences were observed not only in the spatial extent of the activated regions but also in the fMRI signal amplitudes: compared with controls, patient SR showed a profoundly (∼50%) reduced amplitude in response to the vibrotactile stimulation of the contralesional side compared with the ipsilesional side.

Based on these experiments, Beauchamp and Ro concluded that SR's symptom of audiotactile synesthesia was caused by auditory activation of her SII region, which when electrically stimulated evoked a tingling, light touch or subtle electrical sensation in normal human subjects (Penfield and Rasmussen 1950).

Which exact crossmodal plasticity mechanisms may have contributed to the development of synesthesia? Beauchamp and Ro suggested that the stroke-induced lack of somatosensory thalamic input might have allowed short-term unmasking of already existing crossmodal connections between adjacent auditory and somatosensory cortical regions. Based on the delayed onset of the patient's synesthetic symptoms (∼18 mo poststroke), however, the authors assumed a crucial role of long-term plastic changes such as axonal sprouting. Interestingly, this later interpretation is further supported by the authors' earlier discovery of parallel behavioral and anatomical connectivity changes in the same patient (Ro et al. 2007).

In summary, the authors reported a rare and fascinating neurological case. Using fMRI they showed plasticity in the patient's thalamocortical brain circuits that are likely to be the underlying neural basis of her auditory–tactile synesthesia. On the one hand, this study generated a number of important questions regarding the qualitative aspects of SR's acquired synesthesia that we would like to see answered. For example: Does she experience any phenomenological differences between sound- and touch-induced somatosensory sensations? Are there any systematic physical and/or semantic differences between the sounds that induced synesthetic symptoms in SR and noninducing sounds? Investigations of nonacquired synesthesia are currently addressing such questions and framing their answers in the context of lesion studies will further increase our understanding of the neural basis of synesthesia in general.

Tangentially, the findings of Beauchamp and Ro point toward the largely unexplored role of the thalamus in primate multisensory integration. So far, multisensory studies of functional and structural connectivity have emphasized sensory interactions at different cortical levels. Noninvasive studies on potential thalamic contributions to human multisensory integration are largely lacking, in part also reflecting particular sensitivity limitations of the imaging methods and analysis techniques used. Recent advances both in anatomical (e.g., quantitative DTI) and in (multivariate) functional analysis approaches will definitely facilitate future research in this direction. The combination of such increasingly sensitive imaging methods with detailed anatomical knowledge gathered from invasive studies in nonhuman primates (e.g., Cappe et al. 2009) should substantially advance our understanding of how integrated multisensory perception occurs in our brains.

FIG. 1.

Auditory and tactile representations in human thalamus and cortex. A: adjacent and partially overlapping auditory (yellow) and somatosensory (orange and red) representations in the bilateral thalamus and cerebral cortex of healthy human adults (group-averaged fMRI data; n = 16). B: the lesion (black circle) in patient SR's right somatosensory thalamus substantially reduced the ipsilesional thalamic input (orange) to both secondary somatosensory (SII) cortices. Contralesional somatosensory (red) and bilateral auditory input from the thalamus, however, remained unimpaired. Due to this reduction of somatosensory thalamic input opercular SII regions of auditory–somatosensory overlap (striped areas) exhibited altered activation profiles. Whereas somatosensory stimulation led to profoundly reduced activations, auditory responses were found to be substantially enhanced in these SII regions. This most likely forms the neural basis of patient SR's prominent symptom of audiotactile synesthesia, where certain sounds induced intense and often unpleasant somatosensory tingling sensations in her left hand and arm.


We thank J. Mayer and M. Rieder for helpful discussions and C. Bledowski and J. Kaiser for constructive comments on this manuscript.


  1. Beauchamp and Ro 2008.
  2. Cappe et al. 2009.
  3. Driver and Noesselt 2008.
  4. Foxe et al. 2002.
  5. Keysers et al. 2004.
  6. Penfield and Rasmussen 1950.
  7. Ro et al. 2007.
  8. Schroeder et al. 2001.
  9. Stein and Stanford 2008.
View Abstract