Journal of Neurophysiology

Early Experience Determines How the Senses Will Interact

Mark T. Wallace, Barry E. Stein


Multisensory integration refers to the process by which the brain synthesizes information from different senses to enhance sensitivity to external events. In the present experiments, animals were reared in an altered sensory environment in which visual and auditory stimuli were temporally coupled but originated from different locations. Neurons in the superior colliculus developed a seemingly anomalous form of multisensory integration in which spatially disparate visual-auditory stimuli were integrated in the same way that neurons in normally reared animals integrated visual-auditory stimuli from the same location. The data suggest that the principles governing multisensory integration are highly plastic and that there is no a priori spatial relationship between stimuli from different senses that is required for their integration. Rather, these principles appear to be established early in life based on the specific features of an animal's environment to best adapt it to deal with that environment later in life.


The different senses have evolved to work synergistically to enhance the salience of environmental events. This process of multisensory integration is ubiquitous. It has been demonstrated in every species tested, and it is clearly adaptive under widely differing environmental circumstances (Calvert et al. 2004; Stein and Meredith 1993). However, despite its utility it does not appear to be an innate ability because the earliest-appearing multisensory neurons in the superior colliculus (SC) are incapable of synthesizing cross-modal stimuli (Stein et al. 1973; Wallace and Stein 1997, 2001). Rather, they require a protracted period of postnatal maturation to develop this capability, suggesting that sensory experience could be an important determinant. Supporting this is the recent finding that SC neurons in dark-reared animals are unable to integrate visual and nonvisual cues (Wallace et al. 2004).

Presumably, multisensory maturation is protracted because considerable time is needed to gather the experience with combinations of cross-modal cues necessary to link them to common environmental events and to each other. Such coupling cannot be specified in the genome. What appears critical for multisensory interactions is the close temporal and/or spatial correspondence of the different sensory signals derived from common events (Stein and Meredith 1993). Hence, it appears likely that one or both factors are important in dictating which multisensory signals will be integrated and which will not. In this way, sensory experiences can craft a brain to deal with the specific cross-modal contingencies in that animal's world. Although a seemingly reasonable line of logic, there is little empirical evidence to support this view. Strong evidence could be provided if sensory experience was systematically altered during development to disrupt “normal” spatial and/or temporal cross-modal relationships and resulted in corresponding changes in the constraints necessary to elicit multisensory integration. Testing this possibility was the objective of the current study, in which cats were raised with visual and auditory stimuli that were paired in a temporally coincident but spatially disparate manner.


Cats (n = 4) were raised in a modified sensory environment from birth until adulthood (>6 mo). For 12 h each day, the animals were deprived of all visual experiences in a light-tight environment. For the remaining 12 h, a series of controlled visual and auditory stimuli were presented from light-emitting diodes (LEDs) and speakers positioned 1 m from the front of the animal's enclosure. Stimuli consisted of the simultaneous illumination of a single LED and a noise burst from a single speaker. LEDs and speakers were located on separate 4 × 4 grids (16 total locations; individual spacing 0.25 m), and the grids were displaced by 0.75 m relative to one another. Active stimulus pairs were always at the same location on the grid (e.g., 2 over, 3 down), resulting in fixed stimulus separations of 0.75 m (resulting in angular disparities of 18–30° depending on the animal's position). These visual-auditory stimulus pairs were 75 ms in duration and were delivered at random interstimulus intervals (uniform distribution) ranging from 1 to 20 s.

Daily care, observations and routine veterinary procedures were conducted using infrared (IR) goggles and an IR viewing system. Confounding visual experience was precluded when animals were removed from the housing environment for surgery or recording sessions by anesthetizing them in the dark and fitting them with opaque occluding contact lenses and masks during transport and recovery. All data from such spatial-disparity-reared animals were compared with those generated from age-matched animals (n = 3) reared in standard illumination.

All procedures complied with the Guide for the Care and Use of Laboratory Animals (National Institutes of Health Publication 91–3207) at the Wake Forest University School of Medicine, which is accredited by the American Association for Accreditation of Laboratory Animal Care. Details of surgery, recording procedures, and data analyses are similar to those used previously (Meredith and Stein 1983; Wallace and Stein 1997) and are only briefly described.

As adults, each animal was prepared for surgery. Animals were rendered tractable with ketamine (5–25 mg/kg im) and acepromazine maleate (0.2–0.4 mg/kg im) and intubated. Surgical anesthesia was maintained with isofluorane (1.0–4.0%). Throughout the procedure, vital signs (e.g., body temperature, expiratory CO2, etc.) were monitored and maintained at values indicative of deep anesthesia. A craniotomy provided access to the SC and a head-holder was attached to the skull using stainless steel screws and orthopedic cement. Postoperative care (i.e., analgesia, antibiotics) was provided in consultation with veterinary staff.

After 7–12 days of recovery, animals were prepared for recording. They were anesthetized [ketamine (induction: 25 mg/kg im; maintenance: 5–10 mg · kg−1 · h−1 iv)] and paralyzed [pancuronium bromide (2 mg · kg−1 · h−1 iv)] to prevent ocular drift, and infused with fluids [lactated Ringer (8–20 ml/h iv)]. Microelectrodes were advanced into the SC, and single-unit neural activity was recorded, amplified, and routed to an oscilloscope, audio monitor, and computer. Experiments lasted from 6–10 h and were conducted at weekly intervals.

Each SC neuron was classified according to the modality(ies) to which it responded. Receptive fields were mapped using conventional methods and transferred to standardized representations of visual, auditory, and somatosensory space. Receptive field size comparisons were matched by SC location. Receptive field overlap was calculated for visual-auditory neurons and operationally defined as the proportion of the smaller receptive field encompassed by the larger receptive field.

A multisensory neuron was defined as one responding to stimuli from more than one sensory modality or the responses of which to a stimulus in one modality were significantly altered by a stimulus from another modality. Once a neuron's modality selectivity (e.g., visual-auditory) was established, its responses to each modality-specific and cross-modal stimulus combination were quantitatively evaluated. Stimuli were presented 8–20 times at 8- to 12-s intervals in an interleaved fashion. A multisensory interaction was operationally defined as a significant (2-tailed t-test, P < 0.05) change in the response to the combined stimuli when compared with the response to the most effective single-modality stimulus. The magnitude of this interaction was calculated as: (CM − SMmax)/(SMmax) × 100 = % interaction, where CM is the mean response evoked by the combined-modality stimulus and SMmax is the mean response evoked by the most effective single-modality stimulus.

The depth of each recorded neuron was noted in each penetration and electrolytic lesions (5- to 10-μA DC current; 10 s) were made at several locations. Following the final experiment, the animal was killed (sodium pentobarbital, 100 mg/kg iv) and perfused transcardially with saline followed by formalin. The midbrain was blocked stereotaxically, removed, and placed in sucrose overnight. Frozen coronal sections (50 μm) were taken and counterstained (neutral red) to facilitate tissue reconstruction.


Rearing animals in an environment in which brief spatially disparate visual and auditory stimuli were simultaneously presented had little disruptive effect on the distribution of the different sensory-responsive neuronal types in the SC. A significant sample of both unisensory and multisensory neurons was obtained, and the incidence of the various modality convergence patterns was not substantially different from controls (Table 1). Furthermore, although the visual responses of neurons in disparity-reared animals were neither as vigorous nor as well tuned as in controls, in many other respects (e.g., latency, response duration, etc.), they appeared to be similar. They were readily activated by stationary and moving bars of light and had similar variability of response and adaptation characteristics. Outside of an enlargement of receptive fields (see following text), the auditory and somatosensory responses of these neurons were indistinguishable from controls.

View this table:

Distribution of sensory-responsive neuronal types in the SC of spatial-disparity-reared and normally reared animals

Nevertheless, SC neurons were atypical in several respects. First, receptive fields were often so large that they approximated those seen in neonatal and dark-reared animals (Stein et al. 1973; Wallace and Stein 1997; Wallace et al. 2004). Apparently, the minimal visual experience afforded these neurons was insufficient to drive the normal receptive field contraction, a failure that was apparent across all modalities and neuronal types. As a population, visual receptive fields were 61% larger (total of 113 neurons examined), auditory receptive fields 55% larger (n = 76) and somatosensory receptive fields 49% larger (n = 72) than in controls. Despite this, a coarse topography was evident for each sensory representation. An important consideration in these comparisons of receptive field size was that the samples in both the experimental group and the controls were equivalently distributed across the SC (because receptive fields are progressively larger for more caudal SC locations). Histological analyses confirmed an equivalent sampling between these groups.

However, in 39% of the visual-auditory population in which both receptive fields were mapped (a total of 41 neurons), these receptive fields did contract to within 50% of normal adult size. Many of these receptive fields were oddly shaped and exhibited an unusual azimuthal elongation. In contrast to control animals, in which visual receptive fields were only slightly elliptical [mean azimuth:elevation (i.e., aspect) ratio of 1.3:1], these receptive fields had aspect ratios as large as 4:1 and a mean (2.2:1) that was significantly larger (P < 0.01). Although less evident, auditory receptive fields showed a similar azimuthal bias (controls = 1.2:1; experimental animals = 1.6:1, P <. 05). No such bias was evident in somatosensory receptive fields.

The most striking rearing effect was seen in multisensory neurons the receptive fields of which had contracted and become misaligned (Fig. 1, A and B). Receptive field misregister, almost never seen in normally reared cats (Meredith and Stein 1983; Stein and Meredith 1993; Stein et al. 1973; Wallace and Stein 1997), was specific to visual-auditory neurons in spatial-disparity-reared animals. The misregister matched the animal's early sensory experience with auditory receptive fields consistently displaced to the right of the visual receptive field. In the 41 visual-auditory neurons in which both receptive fields could be adequately mapped, the average degree of receptive field overlap was degraded from the control value of 81% to 38% in the experimental group (Fig. 1C; P < 0.01), with a near-continuous distribution of varying degrees of overlap in the experimental population (Fig. 1D). Whereas in controls auditory receptive fields centers were on average 16.0° peripheral to visual receptive field centers, in the spatial-disparity-reared group, the difference was 46.5° (Fig. 1, E and F; P < 0.01). This 30° difference is at the upper end of the range of disparities possible in the rearing environment. Again, these effects were restricted to visual-auditory neurons with visual-somatosensory (81 vs. 79% in controls) and auditory-somatosensory (80 vs. 74% in controls) neurons showing normal overlap.

FIG. 1.

Visual-auditory receptive field overlap is altered in spatial-disparity-reared animals. A and B: 2 examples of receptive fields of visual-auditory neurons in disparity-reared animals. Note the lack of overlap of the visual (blue shading) and auditory (green shading) receptive fields. Receptive fields are plotted on a representation of visual and auditory space, with each 10° shown as a concentric circle and the horizontal and vertical lines indicating 0° azimuth and 0° elevation, respectively. The half circle to the right represents right caudal space (i.e., space behind the interaural axis). C: average amount of visual-auditory receptive field overlap in control (yellow) and spatial-disparity-reared (brown) animals. Error bars represents the SE. D: population plot showing the distribution of visual-auditory neurons with varying degrees of receptive field overlap for the control and experimental populations. E: grand average visual and auditory receptive fields (shading), along with the location of the center of the receptive fields (V, A), for control (left) and spatial-disparity-reared (right) animals. Each concentric circle represents 10°, and the horizontal line shows the horizontal meridian (interaural plane). Grand averages were calculated by taking the horizontal (i.e., azimuthal) and vertical (i.e., elevation) borders of each receptive field, adding these together and dividing by the total number of receptive fields. F. Plot of the displacement of the center of the auditory receptive field relative to the center of the visual receptive field for individual neurons in control (green dots) and spatial-disparity-reared (red dots) animals. Stars show the mean displacement for the two populations. Note the lateral (i.e., peripheral) shift in this index for the experimental population.

The displacement between the visual and auditory receptive fields of multisensory neurons suggests that the spatiotopic sensory organization of the SC was altered by the rearing condition. To examine this possibility, the centroids of both visual and auditory receptive fields were compared in normally and disparity-reared animals for equivalent SC locations. Although there was little discernible change in visual receptive field centroids from normal for a given SC location, Fig. 2 shows that the centroids of auditory receptive fields were peripherally displaced in the disparity-reared population. This result appeared to be preferentially due to alterations in the auditory receptive fields of visual-auditory neurons, which must deal simultaneously with the disparate inputs. However, unisensory auditory neurons and auditory-somatosensory neurons also appeared to have auditory receptive field shifts, albeit of smaller magnitude, but the small sample of such neurons precludes definitive conclusions.

FIG. 2.

The auditory topography in the superior colliculus (SC) appears shifted by spatial-disparity rearing. Left: schematic representation of a dorsal view of the left SC. Numbers within circles correspond to the location of recording penetrations designed to sample from a large extent of the SC. Right: representation of auditory space (conventions are the same as in Fig. 1), with the numbers representing the average auditory receptive field centroid for the recording locations shown on the left for both normally reared (green) and spatial-disparity-reared (red) animals. A, anterior; P, posterior; M, medial; L, lateral; N, nasal; T, temporal; S, superior; I, inferior.

Of particular interest was the effect of the rearing condition on the ability of SC neurons to engage in multisensory integration. This characteristic property did not develop in multisensory neurons whose receptive fields showed little evidence of contraction (i.e., were >200% of control size). Combined visual-auditory stimulation within these large receptive fields elicited responses that were no greater than those to the most effective unisensory stimulus, paralleling the multisensory responses of neonatal SC neurons and of neurons in animals raised in the absence of visual experience (Wallace and Stein 1997; Wallace et al. 2004).

In contrast, many neurons with receptive fields that did contract developed the ability to engage in multisensory integration. Now, however, the visual and auditory stimuli had to be in different locations to fall within the neuron's (misaligned) receptive fields (Fig. 3). To examine the possibility that receptive field contraction serves as a marker for multisensory integrative maturity, the probability of multisensory integration was calculated as a function of relative receptive field size (corrected for SC location). All 13 neurons having receptive fields >200% of control size failed to exhibit multisensory integration; neurons with receptive fields 150–200% of control values had a low probability (P = 0.15, 2/13) and those with receptive fields <150% of controls had a high probability (P = 0.71, 10/14).

FIG. 3.

The spatial constraints of multisensory integration appear dependent on the experiences received during development. Shown are data from 2 neurons, 1 from a control animal (left) and 1 from a spatial-disparity-reared animal (right). Top: receptive fields (visual, blue shading; auditory, green shading) of these neurons, along with the locations of the stimuli used to assess multisensory integration. Conventions are the same as in Fig. 1. Middle: responses of these neurons to visual, auditory, and combined visual-auditory (i.e., multisensory) stimulation. Peristimulus time histograms depict the summed neural responses for a total of 15 trials for each condition with the ramps and square waves at the top showing the timing of the visual and auditory stimulus, respectively. Summary bar graphs at the bottom show the mean responses for each condition, along with the magnitude of the multisensory enhancement (brown bars) and the predicted sum of the visual and auditory responses.

The link among receptive field size, spatial (mis)alignment, and multisensory integration is illustrated in Fig. 4 and highlights the dichotomy between neurons in control and experimental animals. Whereas neurons in control animals had comparatively small receptive fields, a high degree of receptive field overlap and a high probability of exhibiting multisensory integration (Fig. 4A, left), neurons in the spatial-disparity-reared group showed a variable range of receptive field sizes and overlap, but only those with comparatively small receptive fields and little or no overlap had a high probability of exhibiting multisensory integration (Fig. 4A, right). This conclusion is further supported by the multidimensional analysis shown in Fig. 4B and that relates receptive field size, overlap, modality profile (e.g., visual-auditory, visual-somatosensory, etc.), and the presence or absence of multisensory integration in each well-characterized neuron. Note the separation into two distinct clusters of integrative neurons—visual-auditory neurons with small receptive fields and poor overlap and that were likely influenced by the altered spatial experiences and other neurons (nonvisual-auditory) with small receptive fields and good overlap that were presumably not influenced by the altered environment.

FIG. 4.

The development of multisensory integration is dictated by the sensory experiences received during postnatal life, as indexed by changes in receptive field architecture. A: plotted are the percentages of the visual-auditory SC population that exhibit multisensory integration as a function of receptive field overlap for control (left; yellow bars) and spatial-disparity-reared (right; brown bars) animals. In control animals, almost all neurons have a high degree of spatial overlap between their visual and auditory receptive fields, and a high percentage of them exhibit multisensory integration. In contrast, in spatial-disparity-reared animals, many neurons have poor or no receptive field overlap, and it is these neurons that exhibit a high incidence of multisensory integration. Numbers in parentheses above the bars show the total number of neurons for that group. B: plot of receptive field overlap as a function of receptive field size for all sampled neurons in the spatial-disparity-reared population. Each neuron is plotted and categorized as to its modality specificity and integrative capacity (see legend). Note the presence of 2 distinct clusters of integrating neurons (highlighted in gray shading)—on the left visual-auditory neurons with small receptive fields and little or no overlap and on the right nonvisual-auditory neurons with small receptive fields and good overlap.


As noted earlier, the sculpting of multisensory SC receptive fields from their larger neonatal templates and the development of the ability to synthesize multisensory information are gradual processes requiring considerable sensory experience. The present results suggest that the underlying maturational processes are sensitive to the specific details of that cross-modal experience. Rather than being directed toward any predetermined adult profile, they yield a product that reflects the cross-modal activation patterns experienced during early life, even if they seem to violate some principles of the physical world. This effect may be superimposed on many of those processes guiding the maturation of the properties and representations of the individual senses (Buonomano and Merzenich 1998; Feldman 2001; King et al. 2000; Li et al. 2006; Rauschecker 2002; Simons and Land 1987; Singer 1995; Wiesel and Hubel 1965).

The present results suggest that the primary factor in multisensory maturation is the temporal synchrony of different sensory inputs. Presumably, experience with temporally coincident cross-modal stimuli leads to a physiological binding of these stimuli that is reflected in the SC through the creation of spatially overlapping receptive fields and the integrative capabilities of its multisensory SC neurons. In normally reared animals, the most frequent temporally linked cross-modal events are produced by the same events, thereby producing a spatial overlap of the receptive fields of activated multisensory neurons and constraining multisensory enhancement to stimuli that are coincident in space and time (Stein and Meredith 1993). Apparently, this is only one possible developmental outcome. Why some neurons appeared to be insensitive to the particular visual-auditory experiences provided here is unknown. It is, however, quite likely that the stimulus set employed here provided a conservative estimate of the plasticity inherent in the developing SC multisensory representation and that a far greater proportion of its multisensory neurons would have been altered by a contextually richer set of stimuli and/or a stimulus set with immediate behavioral relevance.

Nevertheless, it is interesting to consider the implications of the apparent primacy of temporal synchrony in the maturation of multisensory processes in SC neurons. Multisensory integration changes the physiological salience of an event based on the spatial properties of its cross-modal stimuli: as noted in the preceding text, spatial coincidence among the cross-modal stimuli is normally required for enhanced responses—spatially disparate cross-modal stimuli are either not integrated or result in response depression (Kadunce et al. 1997; Meredith and Stein 1996). Yet the maturation of this integrative capability appears to be an indirect consequence of a neural priority that is entirely unrelated to space. Thus the seemingly odd spatial requirements for multisensory integration in disparity-reared animals are best viewed as the equally direct and logical maturational products of their rearing condition. Whether the temporal synchrony of cross-modal cues proves to be a unique factor for coupling other sensory modalities during early maturation, and/or is as powerful a determinant of multisensory integration in other brain areas, remains to be determined.

It is also important to note that there were no compensatory auditory receptive field shifts seen here, such as those seen after rearing animals with deviations of the visual axis (King et al. 1988; Knudsen and Brainard 1991, 1995). These studies have shown that the auditory system uses vision to calibrate its map in the SC and that spatial discrepancy between the maps can be minimized by shifting the auditory receptive fields to better realign with their visual counterparts. As a result, the misalignments among the different receptive fields of multisensory neurons are also minimized. Although not tested in these previous studies, the present data strongly suggest that multisensory integration would develop in these circumstances and would accurately reflect the realignment of receptive fields. There is little reason to doubt that cross-modal temporal coincidence is responsible for this compensatory process as well. Presumably, however, the compensatory shifting of auditory receptive fields also requires feedback from the motor acts involved in dealing with the initiating events. These instructive signals were unavailable to the brain in the present experiments, thereby revealing an underlying cross-modal process that was not “corrected” by the initiation of other adaptive processes. Whether interactive experience with aligned cross-modal cues during adulthood could bring the receptive fields into register and adapt the multisensory integrative capabilities of these multisensory SC neurons remains to be determined.


This work was supported by National Institutes of Health Grants MH-63861 and NS-36916.


We thank N. London for editorial assistance.


  • The costs of publication of this article were defrayed in part by the payment of page charges. The article must therefore be hereby marked “advertisement” in accordance with 18 U.S.C. Section 1734 solely to indicate this fact.


View Abstract