JN Fuel your research with LabChart
HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS
 QUICK SEARCH:   [advanced]


     


J Neurophysiol 97: 1738-1755, 2007. First published November 8, 2006; doi:10.1152/jn.00189.2006
0022-3077/07 $8.00
This Article
Right arrow Abstract Freely available
Right arrow Full Text (PDF)
Right arrow All Versions of this Article:
97/2/1738    most recent
00189.2006v1
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in ISI Web of Science
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Download to citation manager
Citing Articles
Right arrow Citing Articles via HighWire
Right arrow Citing Articles via ISI Web of Science (5)
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Merriam, E. P.
Right arrow Articles by Colby, C. L.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Merriam, E. P.
Right arrow Articles by Colby, C. L.

Remapping in Human Visual Cortex

Elisha P. Merriam1,3, Christopher R. Genovese2,3 and Carol L. Colby1,3

1Department of Neuroscience, University of Pittsburgh; 2Department of Statistics, Carnegie Mellon University; and 3Center for the Neural Basis of Cognition, Pittsburgh, Pennsylvania

Submitted 21 February 2006; accepted in final form 1 November 2006


    ABSTRACT
 TOP
 ABSTRACT
 INTRODUCTION
 METHODS
 RESULTS
 DISCUSSION
 APPENDIX
 GRANTS
 ACKNOWLEDGMENTS
 REFERENCES
 
With each eye movement, stationary objects in the world change position on the retina, yet we perceive the world as stable. Spatial updating, or remapping, is one neural mechanism by which the brain compensates for shifts in the retinal image caused by voluntary eye movements. Remapping of a visual representation is believed to arise from a widespread neural circuit including parietal and frontal cortex. The current experiment tests the hypothesis that extrastriate visual areas in human cortex have access to remapped spatial information. We tested this hypothesis using functional magnetic resonance imaging (fMRI). We first identified the borders of several occipital lobe visual areas using standard retinotopic techniques. We then tested subjects while they performed a single-step saccade task analogous to the task used in neurophysiological studies in monkeys, and two conditions that control for visual and motor effects. We analyzed the fMRI time series data with a nonlinear, fully Bayesian hierarchical statistical model. We identified remapping as activity in the single-step task that could not be attributed to purely visual or oculomotor effects. The strength of remapping was roughly monotonic with position in the visual hierarchy: remapped responses were largest in areas V3A and hV4 and smallest in V1 and V2. These results demonstrate that updated visual representations are present in cortical areas that are directly linked to visual perception.


    INTRODUCTION
 TOP
 ABSTRACT
 INTRODUCTION
 METHODS
 RESULTS
 DISCUSSION
 APPENDIX
 GRANTS
 ACKNOWLEDGMENTS
 REFERENCES
 
As the eyes move, objects in the world change position on the retina. Despite the constant displacement of retinal images, a coherent and stable visual scene is perceived. This phenomenon, termed spatial constancy, indicates that the brain constructs a stable representation of the visual world by combining information about voluntary eye movements with sensory information from the visual system. Neurophysiological evidence for updating has emerged in recent years. Single-unit recording studies indicate that neurons in monkey lateral intraparietal cortex (area LIP) update, or remap, the locations of salient stimuli in conjunction with voluntary eye movements (Duhamel et al. 1992aGo; Goldberg et al. 1990Go; Gottlieb et al. 1998Go; Heiser et al. 2005Go; Kusunoki et al. 2000Go). In these studies, the monkey fixates and a stimulus is briefly presented outside the receptive field of the neuron being recorded. When a new fixation point appears, the monkey makes an eye movement to it. This eye movement brings the receptive field onto the recently stimulated screen location. LIP neurons become active at this time even though there is no physical stimulus in the receptive field. This activity is interpreted as a response to the memory trace of the brief visual stimulus. Updating depends on a corollary discharge of the eye-movement command signal (Sommer and Wurtz 2002Go). The response to the updated trace of the stimulus indicates that LIP neurons perform a coordinate transformation that enables the visual system to maintain perceptual continuity across saccades (Colby and Goldberg 1999Go).

Parietal cortex is centrally important in creating an updated representation of space. The majority of LIP neurons exhibit remapping (Duhamel et al. 1992aGo), and reversible inactivation of LIP impairs performance on tasks that require updated spatial information (Li and Andersen 2001Go). Studies in humans also indicate a central role for parietal cortex. Neurological patients with parietal lobe lesions are impaired on tasks that require spatial updating (Duhamel et al. 1992bGo; Heide et al. 1995Go; Khan et al. 2005Go). Imaging studies have also demonstrated the role of parietal cortex in remapping. We have used fMRI to show that representations of visual stimuli are updated from one hemisphere to the other in conjunction with horizontal single-step saccades (Merriam et al. 2003Go). Similarly, several fMRI studies have used double- and triple-step saccade tasks to demonstrate remapping in humans (Heide et al. 2001Go; Medendorp et al. 2003Go, 2005aGo,bGo). These studies indicate that there is a functional similarity between the computations performed by parietal neurons in monkeys and humans (Crawford et al. 2004Go; Merriam and Colby 2005Go).

Remapping activity is not limited to parietal cortex. Remapping has been observed in the frontal eye field (FEF), the superior colliculus (SC), and extrastriate visual cortex. Neurons in all these areas have spatially selective visual and perisaccadic responses, are modulated by spatial attention, and respond to the stimulus trace in the single-step saccade task (Nakamura and Colby 2002Go; Umeno and Goldberg 1997Go, 2001Go; Walker et al. 1995Go). If remapping is important for perceptual constancy, remapping should not be limited to brain regions with attentional and oculomotor functions. Rather, updated spatial information should reach visual areas that are involved in visual perception. The goal of the present study was to test the hypothesis that updating occurs in early visual cortex in humans. Two lines of evidence suggests that it does. First, psychophysical studies have demonstrated that updated visual signals are required to integrate information about stimulus features across saccades (Hayhoe et al. 1991Go; Melcher and Morrone 2003Go; Prime et al. 2006Go). Second, several human fMRI studies have demonstrated strong top-down effects throughout occipital cortex. Multiple visual areas are activated in tasks that involve spatial attention (Brefczynski and DeYoe 1999Go; Gandhi et al. 1999Go; Kastner et al. 1999Go; McMains and Somers 2004Go; Ress et al. 2000Go; Silver et al. 2005Go; Tootell et al. 1998Go; Yantis et al. 2002Go). Many of these areas are also modulated by oculomotor signals (DeSouza et al. 2002Go; Sylvester et al. 2005Go). These fMRI studies indicate that visual cortex has access to the corollary discharge signals necessary for remapping.

We used an fMRI version of the single-step task to test whether remapped visual signals are present in visual cortex. In this task, subjects fixate while a salient visual stimulus flickers in the periphery. The stimulus is expected to activate visually responsive cortex in the contralateral hemisphere. The stimulus is then extinguished, and a tone cues the subject to make an eye movement to a stable target. The target position is chosen so that the eye movement brings the location of the now-extinguished stimulus into the opposite hemifield. The premise of this experiment is that activation related to the memory trace of the stimulus is remapped from one hemisphere to the other with the eye movement. We predicted that the hemisphere that was initially ipsilateral to the stimulus would become active around the time of the eye movement. We found strong evidence for remapping in striate cortex and in each extrastriate visual area examined. Further, we found that remapping was more robust in higher-order extrastriate areas. Our results indicate that remapping is present in visual areas that are directly linked to visual perception.


    METHODS
 TOP
 ABSTRACT
 INTRODUCTION
 METHODS
 RESULTS
 DISCUSSION
 APPENDIX
 GRANTS
 ACKNOWLEDGMENTS
 REFERENCES
 
Subjects

We studied a total of 14 healthy participants (7 female, aged 25–35). All subjects had extensive prior experience with both fMRI and psychophysical experiments. Informed written consent was obtained in accordance with the University of Pittsburgh IRB. All subjects had normal or corrected vision. Data from two subjects were discarded because of noise artifacts in the MR data.

Behavioral paradigms

Visual stimuli were generated on a PC computer using the Psychophysics Toolbox (Pelli 1997Go) running in Matlab 6.5 (Mathworks, Natick, MA). Stimuli were presented via an LCD projector and long-throw optics onto a back-projection screen in the bore of the MR scanner. Subjects viewed the projected stimuli through an angled mirror, resulting in a 10° vertical x 20° horizontal field of view. We measured fMRI activation while subjects performed three tasks, as described in the following text.

SINGLE-STEP TASK. Two stable crosses were located 8° to the left and right of screen center. Subjects fixated one of the two crosses at the beginning of the trial (Fig. 1A). After a variable fixation period (1,000 ± 200 ms), a small (1–2°) visual stimulus appeared at the center of the screen and 3° above the horizontal axis in the upper quadrant of the right or left visual field. Subjects were instructed to maintain fixation on the cross and not to look at the stimulus. After 1 s, the stimulus was extinguished, and a tone cued the subject to make an eye movement to the opposite fixation cross. This saccade brought the screen location of the now-extinguished stimulus into the opposite visual field. The trial ended after a variable period of fixation (2,000 ± 200 ms) when a second tone instructed the subject to make a return saccade back to the initial fixation cross.


Figure 1
View larger version (28K):
[in this window]
[in a new window]

 
FIG. 1. Three task conditions. A: single-step task. Subject fixates a cross (FP1) located 8° to the left of screen center. After a variable period (1,000 ± 200 ms), a salient stimulus appears in the right visual field and flickers on the screen for 1 s. The stimulus (full circles) activates contralateral (left) hemisphere visual cortex. Next, the stimulus is extinguished and a tone cues the subject to make a rightward eye movement to FP2. This saccade brings the screen location of the now-extinguished stimulus (empty circles) into the left visual field. After a variable period (2,000 ± 200 ms), a 2nd tone instructs the subject to make a return saccade back to FP1. B: stimulus-only fixation task. Subject fixates FP1. After a variable period (1,000 ± 200 ms), the visual stimulus appears in the periphery. The stimulus is extinguished after 1,000 ms. The trial ends after a variable fixation period (2,000 ± 200 ms). C: saccade-only control task. Subject fixates FP1. After a variable period (2,000 ± 200 ms), a tone cues the subject to make an eye movement to FP2. The subject fixates for a variable period (2,000 ± 200 ms) until a 2nd tone cues them to make a return saccade. Dashed squares and arrows indicate the location of the eyes.

 
STIMULUS-ONLY FIXATION TASK. The visual events in the stimulus-only fixation task were similar to those in the single-step task. Two stable crosses were located 8° to the left and right of screen center. Subjects fixated one of the two crosses at the beginning of the trial. After a variable period (1,000 ± 200 ms), a small (1–2°) visual stimulus appeared at the center of the screen, 3° above the horizontal meridian in the upper quadrant of the right or left visual field. Subjects continued to fixate while the stimulus flickered on the screen for 1 s. The trial ended 2,000 ± 200 ms after the stimulus was extinguished. Subjects maintained fixation for the entire duration of the trial. There are two versions of the stimulus-only fixation task: an ipsilateral and a contralateral version. The ipsilateral version of the task is similar to the single-step task in that the stimulus appears in the ispilateral hemifield. The only difference between the tasks is that no auditory cue is presented and subjects do not make a saccade (Fig. 1B). We used the ipsilateral version of the task as a control condition in several analyses. In the contralateral version of the task, the stimulus was located in the contralateral visual field. We used this version of the task to identify visually responsive voxels.

SACCADE-ONLY TASK. This task is similar to the single-step task. The only difference is that no salient visual stimulus appears prior to the eye movement (Fig. 1C). Two crosses were located 8° to the left and right of screen center. Subjects fixated one of two crosses at the beginning of the trial. After a variable period (2,000 ± 200 ms), a tone cued the subject to make an eye movement to the opposite fixation cross. Subjects maintained fixation for a variable period (2,000 ± 200 ms) until a second tone instructed them to make a return saccade back to the initial fixation cross. The timing of the task was identical to the single-step saccade task.

The stimulus-only fixation task and saccade-only task were intended to control for sensory and motor factors of the single-step task that are not specific to remapping. For example, saccades in both the single-step task and the saccade-only task were triggered by an auditory cue. Activation attributable to the auditory stimulus should thus be equal in the two tasks.

Trials of each condition lasted an average of 4,000 ms, including the variable fixation period. We found that varying the length of the initial fixation period from trial to trial reduced the number of instances in which subjects made an eye movement prior to the auditory cue. As a consequence of this variability in fixation duration, the start time of each trial was not yoked to the scanner TR. We recorded trial timing information, eye position, and scanner pulses using custom software. We then used this timing information in the analysis of the fMRI data.

Experimental design

Each subject participated in at least two scanning sessions, one for the main remapping experiment, and another session for identifying the location of each visual area using retinotopic mapping procedures (described in the following text). A single scanning session lasted ~1.5 h; each session consisted of 6–10 runs; each run lasted 512 s; 64 trials were tested in each run. On a subset of runs (3–5), all trials began with fixation on the left cross, and on the other subset of runs, trials began with fixation on the right cross. Each of the three tasks was performed in both directions over the course of the session. This was a critical feature of the experimental design because it enabled us to measure responses in each hemisphere when the stimulus was located in either the contralateral or ipsilateral visual field. The two initial fixation positions were never mixed within a run: on runs in which the stimulus appeared in the left visual field, the stimulus never appeared in the right visual field. During the course of a scanning session, subjects performed 128–256 repetitions of each task.

We used an experimental design in which periods of fixation were interspersed with experimental trials. During the periods of fixation, or null trials, subjects simply maintained fixation on the initial cross for 4,000 ms. Periods of fixation were matched with experimental trials for orbital position, duration, and frequency. The ordering of experimental and null trials was determined by a special class of pseudorandom sequences known as m-sequences (Reid et al. 1997Go; Sutter 2001Go). Randomly generated stimulus sequences often have temporal autocorrelations that can interfere with response estimation. In contrast, m-sequences have a nearly-flat autocorrelation function. This property makes m-sequences advantageous for fMRI (Buracas and Boynton 2002Go; Liu 2004Go; Liu and Frank 2004Go).

Eye-position recording

We monitored eye position during each fMRI session using a video-based eye tracker (ASL, Boston, MA). The eye tracker had a temporal resolution of 60 Hz. Stability of eye data was typically better than 2° as determined by the SD of the data during periods of stable fixation. Figure 2, A–C, shows eye traces from a single subject recorded during scanning.


Figure 2
View larger version (12K):
[in this window]
[in a new window]

 
FIG. 2. Analysis of eye-position data. Eye traces from 64 trials of the single-step task (A), the saccade-only control task (B), and the stimulus-only control task (C). Calculated saccadic reaction time (SRT) is indicated by tick marks in A and B. This subject made 1 anticipatory saccade in the single-step task and 1 late saccade in the saccade-only task. Error trials were excluded from the analysis of fMRI data. D: no difference in saccade reaction time in trials of the single-step task (light gray bars) and saccade-only control task (dark gray bars). Saccadic reaction times across trials are represented using standard box-and-whisker plots (vertical lines at the center of each box indicate the median; box-ends indicate the quartiles).

 
Analysis of the eye position data ensured that subjects maintained stable fixation within a 2° window on FP1 during the 1 s of visual stimulation, made horizontal eye movements in a 500-ms temporal window after the auditory cue, and made accurate saccades (to within 2°) on single-step and saccade-only trials. We analyzed each subject's eye position data on a trial-by-trial basis and discarded trials in which subjects failed to meet this set of performance criteria. Subjects made errors on <5% of trials.

We used an automatic saccade finding algorithm to search for saccades in the first two seconds of each trial [http://groups.yahoo.com/group/ilab (Gitelman 2002Go)]. The software identified the occurrence of a saccade if eye velocity exceeded 50 °/s and the eyes moved >2°. Subjects were expected to have made saccades in response to the auditory cue on single-step and saccade-only trials. Mean saccadic reaction time was 255 ± 128 (SD) ms in the single-step task and 269 ± 136 ms in the saccade-only condition. The difference in saccadic reaction time was not significant (t-test, P > 0.05). It was critical to our experiment that subjects not make a saccade while the stimulus was still present on the screen. On occasional trials, subjects made an early saccade, while the stimulus was still present and prior to the auditory cue. These early-saccade trials were not included in the analysis of the fMRI data.

MRI data acquisition and preprocessing

We used functional magnetic resonance imaging at 3T (Allegra, Siemens, Erlangen) and a T2*-sensitive EPI pulse sequence to measure changes in BOLD activity. Scan parameters were as follows: TR = 1,000 ms; TE = 30 ms; flip angle = 65°. We collected 16 slices (3 mm3 voxels, 192 mm field of view) in each volume, and 512 volumes in each functional run. Slices were oriented perpendicular to the calcarine sulcus to cover the entire occipital lobe (Fig. 3A). Functional data were preprocessed using FIASCO software [http://stat.cmu.edu/~ fiasco (McNamee and Eddy 2001Go)]. Preprocessing steps included correction for fluctuations in mean intensity; motion correction of the raw k-space data (Eddy et al. 1996Go); image reconstruction, and outlier correction using a Windsor filter. Outliers were defined as data points farther than 10 times the interquartile range from the median. The reconstructed MR images were not smoothed, temporally filtered, or spatially normalized.


Figure 3
View larger version (52K):
[in this window]
[in a new window]

 
FIG. 3. A: 16 slices were oriented perpendicular to the calcarine sulcus. Slices were 3 mm thick and covered the entire occipital lobe. B: 3-dimensional (3D) rendering of the cortical surface at the gray/white matter boundary. Shaded gray region indicates the cortical region of interest, shown as a flattened patch in the four panels below. Each disk in C–F shows the same flattened patch of cortex with a 50-mm radius. Gray background represents estimated cortical curvature: (dark gray, concave; light gray, convex). C: representation of polar angle. The stimulus was a rotating 15° checkerboard wedge. This map was used to define borders between the ventral portions of V1, V2, V3, and hV4. D: representation of visual eccentricity. The stimulus was an expanding/contracting 15° checkerboard annulus. In both C and D, hue represents the stimulus location that elicited the maximal response. E: response to contralateral visual stimulus in fixation task. F: response to the updated stimulus trace in the single-step task. In both E and F, hue (red–yellow) represents the magnitude of the response. Color opacity (transparent–opaque) represents the probability that the response to the stimulus was nonzero. Opacity values range from 0 to 1; the data are not thresholded.

 
Statistical modeling

We used a fully Bayesian approach to analyzing the MR data, the details of which have been described elsewhere (Genovese 1998aGo,bGo, 2000Go) (see APPENDIX). Briefly, we fit the fMRI time series data with a nonlinear, hierarchical model that decomposes the observed signal into four components: baseline, drift, activation, and noise. The activation component is further subdivided into lag, attack, and decay. We used Bayesian statistical methods to derive the posterior distribution of the parameters given the data, P({gamma}|Y). From this distribution, we computed posterior probabilities related to our questions of interest as well as point estimates (posterior means) and their posterior SDs. All inferences in this study are based on these posterior distributions of the parameter vector through derived posterior quantities. This analysis yields a Bayesian posterior probability, which we denote as "q." The posterior probability should not be confused with a P value from a classical statistical test (i.e., the probability that the data could be drawn from the population tested given the assumption that the null hypothesis is true).

We make four key inferences in this study. The first is the probability that there is a nonzero response in a task condition. For condition "c," this is denoted by p{{gamma}cresp > 0|Y}, where {gamma}cresp is the response magnitude parameter of the model. Because our hierarchical model allows for a nonzero probability on the discrete value 0, this probability indicates the strength of evidence for a response above baseline. The second inference that we consider is the probability that the response magnitude in one condition, c, is greater than the response magnitude in another, c', denoted by P{{gamma}cresp > {gamma}c'resp|Y}. We make similar comparisons for shape parameters, such as when comparing response onset times across conditions, denoted by P{{gamma}cttp > {gamma}c'ttp|Y}. Third, we make inferences regarding more complex events, such as the posterior probability that the response in one condition, c, is larger than the combined responses in two other conditions, c' and c'', denoted by P{{gamma}cresp > ({gamma}c'resp + {gamma}c''resp)|Y}. Finally, we make inferences about the population by combining the voxel-wise point estimates across voxels in a visual area in a single hemisphere, and across hemispheres and subjects, yielding p(Formula|Y). This enables us to make group-level comparisons across visual areas. For example, we compare the combined size of a response in a given cortical area, a, to that in a another area, a', denoted by P{Formulaaresp > Formulaa'resp|Y}.

Hierarchical Bayesian statistical models confer numerous advantages over models that use classical statistical procedures (for discussion, see Genovese 2000Go). The use of Bayesian statistics has become increasingly common in fMRI data analysis and a number of Bayesian statistical models have been described in the literature (see Friston and Penny 2003Go; Friston et al. 2002aGo,bGo; Genovese 2000Go; Gössl et al. 2001Go; Marrelec et al. 2003Go, 2004Go; Penny et al. 2003Go, 2005Go; Smith et al. 2003Go; Woolrich et al. 2004aGo,bGo).

Voxel selection

We used three criteria to select functional voxels. First, we identified gray-matter voxels. Second, we selected voxels based on inclusion within the boundaries of predefined cortical visual areas and omitted voxels that straddled borders. Third, we selected the subset of voxels that responded to the small and brief contralateral stimulus in the stimulus-only fixation task. Each of these criteria are described in more detail in the following text.

GRAY-MATTER SEGMENTATION. We acquired two or three three-dimensional (3D) anatomical volumes from each subject using an MPRAGE pulse sequence (30 ms TE, 8° flip angle, 192 slices, 1 mm3 voxel resolution) and averaged the volumes to increase the signal to noise ratio. Whole-brain 3D anatomical images for each subject were registered to that subject's functional data using a fully automated algorithm implemented in FIASCO software. Gray matter, white matter, and CSF were then segmented using FreeSurfer software http://surfer.nmr.mgh.harvard.edu (Fischl et al. 1999Go); http://surfer.nmr.mgh.harvard.edu (Dale et al. 1999Go). We used the estimated boundary between white and gray matter to select functional voxels for further analysis. We visualized activation on 2D flattened representations of the cortical surface (Fig. 3, CF). Flat maps were created using the mrVista MATLAB toolbox [http://white.stanford.edu/software (Wandell et al. 2000Go)].

RETINOTOPIC SPECIFICITY. We used standard retinotopic mapping procedures to identify the borders between occipital lobe visual areas V1, V2, V3, V3A, and hV4 (DeYoe et al. 1996Go; Engel et al. 1994Go, 1997Go; Sereno et al. 1995Go). Retinotopic mapping was carried out in a separate scan session. Subjects underwent six to eight runs of eccentricity and polar angle mapping (Fig. 3, C and D). We used retinotopic mapping stimuli that moved smoothly across visual space, creating a traveling waves of cortical activity. Stimuli were contrast and hue-modulated flickering checkerboards that took the shape of rotating wedges and expanding/contracting annuli. The details of the stimuli have been reported elsewhere (Tootell et al. 1997Go). The spatial frequency of the checkerboards was scaled to accommodate larger receptive fields in the periphery. Stimulus movement was periodic, with a frequency of 1/64 s. The stimulus completed 8.5 cycles/run. We used the phase cancellation technique described by Kalatsky and Stryker in which the direction of stimulus movement was reversed on successive runs. We summed the complex-valued data prior to calculating the phase and magnitude of the response. This procedure removes the hemodynamic delay associated with the blood-level-oxygen-dependent (BOLD) response, thereby yielding more accurate estimates of the stimulus position that elicited the maximal response.

The retinotopic mapping stimuli encompassed a large portion of the visual field (20° horizontal x 15° vertical) and hence activated a broad region of cortex (Fig. 3, C and D). Visual area boundaries were defined using a conjunction of polar angle and eccentricity maps according to the following three criteria, as described by Dougherty et al. (2003)Go. First, each area was bounded by phase reversals in the angular component of the retinotopic map. Second, a given area had to be activated by both the wedge and annulus stimuli. Third, the phase gradient in the angular and eccentricity maps had to run orthogonal to one another. Both the dorsal and ventral portions of areas V1, V2, and V3 were easily identified using these criteria.

We identified area hV4 using the criteria described by Tootell and Hadjikhani (2001)Go. Area hV4 is the ventral retinotopic area that continues laterally from ventral V3. Area hV4 has a full hemifield representation and is located proximal to the medial lip of the collateral sulcus. We refer to this area as "hV4" because the degree of functional correspondence between this region and monkey area V4 has not been fully resolved (Brewer et al. 2005Go; Tootell and Hadjikhani 2001Go). It is possible that the cortical area that we labeled hV4 also contains additional subdivisions (e.g., areas VO-1 and VO-2) (Brewer et al. 2005Go).

We identified area V3A using the criteria described by Tootell et al. 1997Go Area V3A is the dorsal retinotopic area that continues anteriorly from dorsal V3. Area V3A contains a full hemifield representation and is located proximal to the transverse occipital sulcus at the base of the intraparietal sulcus. It is possible that the cortical area that we identified as V3A also contained other cortical areas (e.g., V3B) (Press et al. 2001Go). Area V7 was not reliably identified in our data and was therefore not included in this study.

Of the 24 hemispheres in this experiment, all but 2 had clean retinotopic maps in which the borders between visual areas could be determined unambiguously. In two hemispheres (1 left, 1 right) from two different subjects, the borders between visual areas were not clearly discernible. These hemispheres were not included in the analysis. Hence, all analyses report results from n = 22 hemispheres from 12 participants.

SELECTING VOXELS BASED ON VISUAL RESPONSIVENESS. In the version of the single-step task used in neurophysiology experiments, an eye movement brings the receptive field of the neuron onto the recently stimulated screen location (Duhamel et al. 1992aGo). Remapped activity is observed as a response to the memory trace of the stimulus in the absence of any direct visual stimulation. Implicit in the logic of this paradigm is the assumption that the neuron fires when an actual stimulus appears inside of its receptive field. The goal of this third voxel selection procedure was to identify the set of voxels that respond to the visual stimulus used in this experiment—these are the same voxels that we predict could also exhibit remapping.

In both the single-step task and the stimulus-only fixation task, a small (1–2°) stimulus appears in the upper visual field for 1 s (Fig. 1, A and B). This stimulus should activate only a subset of the voxels that are activated by the large checkerboard stimuli in the retinotopic mapping experiment. We identified this subset of voxels by analyzing responses in the stimulus-only fixation task when the stimulus appeared in the contralateral visual field. For each voxel, we estimated the magnitude of the visually evoked response, {gamma}visualresp, and the posterior probability that this magnitude was greater than zero given the data, P{{gamma}visualresp > 0|Y}. Results from this analysis are plotted on the cortical surface (Fig. 3E). In this plot, activation magnitude is represented by a red–yellow color scale: voxels with large visual responses are yellow and voxels with smaller visual responses are red. Posterior probability is represented by color opacity with zero probability being fully transparent; no statistical threshold has been applied to these results. Yellow voxels tend to be more opaque because large responses tend to have a higher probability of being nonzero.

We measured visual responses in the stimulus-only fixation task when the stimulus appeared in the contralateral visual field. These visual responses had two notable properties (Fig. 3E). First, voxels activated by the visual stimulus were located in the appropriate region of each retinotopic map. In the fixation task, the stimulus appeared 3° above the horizontal axis and 8° from the vertical axis, in the upper quadrant of the right or left visual field. We thus expected the stimulus to activate the contralateral upper visual field representation at ~9° eccentricity. This location in cortex is indicated by shades of magenta in the polar angle map (Fig. 3C) and by shades of yellow/green in the eccentricity map (Fig. 3D). These activation maps indicate that visual responses in the fixation task were located in the subregion of each visual area that correspond to the appropriate location in the retinotopic map.

Second, there is a clear distinction between active and inactive voxels. Posterior probabilities tended to be either high (far greater than chance, q ≥ 0.5), or low (far less than chance, q << 0.5). Because of this property, voxels appear as either fully opaque or completely transparent (q approaches 0 in cortical locations in which the underlying grayscale anatomy is clearly visible). All subsequent analyses were performed on the subset of voxels in each area that met a q greater double equals 0.95 selection criteria for contralateral visual stimuli. The sharp distribution of probability values indicates that the particular threshold did not have a strong impact on which voxels were included in the analysis.

Response normalization

There was considerable variability across subjects and visual areas in the magnitude of the visual response in the stimulus-only fixation task. It is not clear whether this variability is due to differences in neural response strength or to interregional and intersubject differences in hemodynamics. Logothetis and Wandell (2004)Go have argued that regional differences in the coupling between neural activity and hemodynamic changes could result in spurious differences in response magnitude across cortical areas. They termed this coupling hemodynamic response efficiency or HRE. It is common to normalize MR activation by the baseline signal level on a voxelwise basis, thereby expressing activation in units of percent signal change. However, normalizing by the baseline does not account for regional differences in HRE. One solution to this problem is to normalize responses by the magnitude of activation in a second condition that is known to elicit a consistent response. Such selectivity measures reflect the proportional increase or decrease in activation in a particular area, given the regional HRE. The data in this experiment lend themselves to this normalization procedure because voxels were selected based on there being a highly probable visual response. Based on this line of reasoning, we normalized activation magnitudes in each condition by the magnitude of the visual responses. Normalized values were calculated as follows. We first identified the set of visually responsive gray matter voxels in a given visual area for a given hemisphere as described in the preceding text. We then averaged visual responses across voxels within a given visual area, which we call Formulavisualresp, and responses from the condition of interest, Formulacresp. Finally, we took the ratio of the two means

Formula 1(1)
As a result of this normalization procedure, all response magnitudes are expressed as a percent of the visual response, rather than as a percent of the baseline MR signal.


    RESULTS
 TOP
 ABSTRACT
 INTRODUCTION
 METHODS
 RESULTS
 DISCUSSION
 APPENDIX
 GRANTS
 ACKNOWLEDGMENTS
 REFERENCES
 
Our central finding is that cortical visual areas ipsilateral to the stimulus respond in the single-step task. These responses cannot be attributed to either the ipsilateral stimulus alone or to saccades alone. We interpret this activation as a response to the trace of the stimulus that has been remapped from the contralateral to the ipsilateral hemisphere with the saccade. We begin by illustrating this result with responses from a single right hemisphere hV4 voxel (Fig. 4). This single-voxel is representative of the larger population of voxels from the 22 hemispheres in this study. In subsequent sections, we combine data across hemispheres and subjects to show that the main findings illustrated in this example voxel are characteristic of the larger population.


Figure 4
View larger version (25K):
[in this window]
[in a new window]

 
FIG. 4. Hemodynamic responses in a single right hemisphere hV4 voxel that exhibits remapping. The cartoon in each panel shows the location of the stimuli on the screen. Horizontal eye position and timing of stimulus events are shown below (calibration bar, 16°). A: visual response in fixation task. A contralateral stimulus during fixation elicits a strong response. B: remapped response in single-step task. The subject fixates FP1 as a stimulus flickers in the ipsilateral visual field for 1 s. After 1 s, the stimulus is extinguished and a tone cues the subject to make a saccade to FP2. The saccade brings the screen location of the extinguished stimulus into the contralateral visual field. The remapped trace of the stimulus elicits a response. {uparrow}, onset time of the visual (- - -) and remapped response (—). C: stimulus-only control. Presentation of the stimulus in the ipsilateral visual field does not elicit a response in the absence of a saccade. D: saccade-only control. The saccade alone does not elicit a strong response in the absence of a stimulus. Each curve was estimated from responses on 64 trials. {square}, 1 SE.

 
We refer to the MR response in this condition as a visual response because it is driven by direct, contralateral visual stimulation. Each response curve in Fig. 4 represents the averaged response from 64 trials per condition. This voxel is strongly activated by the appearance of a stimulus in the contralateral visual field during the fixation task (Fig. 4A). In contrast, the voxel did not respond strongly when a stimulus was presented in the ipsilateral hemifield during fixation (Fig. 4C) nor did it respond in conjunction with a saccade from FP1 to FP2 when no stimulus was present (Fig. 4D). The critical finding is that this voxel did respond after the screen location where the stimulus had recently appeared was brought into the contralateral visual field by a saccade (Fig. 4B). We interpret this activity as a response to the remapped memory trace of the stimulus. The eye traces for each condition are shown in the following text. We calculated saccade latency on each of 64 trials of the single-step task from eye-position data recorded during scanning (Fig. 4B, eye traces). This subject had an average saccadic reaction time of 257 ± 55 ms. Analysis of the eye-position data confirms that the eyes began to move only after the stimulus had already been extinguished. Because the stimulus was never physically present in the contralateral visual field, we conclude that this voxel responded to the remapped trace of the stimulus.

The temporal profile of the response on single-step trials also indicates that the response is driven by the remapped trace of the stimulus rather than by the stimulus itself. In the single-step task, the stimulus appears and stays on the screen for 1,000 ms prior to the cue to make an eye movement. The eye traces indicate that the eyes began to move ~200 ms after the tone. The response to the remapped stimulus trace should have a latency that is ~1,200 ms longer than the response to the visual stimulus. The example voxel in Fig. 4 indicates that this is in fact the case. The visual response in Fig. 4A begins to rise at ~2,000 ms after the onset of the stimulus, consistent with the time course of visually-driven hemodynamic response curves (Boynton et al. 1996Go). In contrast, the remapped response in Fig. 4B begins to rise at ~3,000 ms after the onset of the stimulus. The latency difference between these two curves corresponds to the period between the onset of the visual stimulus and the auditory cue to initiate a saccade.

In summary, the example voxel illustrates four response properties that characterize remapping. This voxel responded to the contralateral visual stimulus in the fixation task, responded to the remapped stimulus trace in the single-step task, did not respond strongly in either of two control conditions, and responded in the single-step task at a latency predicted by the timing of the task. We characterize each of these four response properties below using a Bayesian statistical model of the fMRI time series data. The goal of this analysis is to quantify the degree to which these four properties are present in each visual area across the group of subjects.

Responses to the remapped stimulus trace

The central question in this experiment is whether early- and intermediate-level visual areas respond to the remapped trace of the stimulus. As part of our voxel selection criteria, we identified voxels that responded to direct, contralateral visual stimulation. In this section, we ask whether these same voxels also respond to the remapped trace of a stimulus presented in the ipsilateral visual field (Fig. 1A). Remapped responses were measured on trials of the single-step task in which the stimulus appeared in the ipsilateral visual field and a subsequent eye movement brought the recently stimulated screen location into the contralateral visual field. For each voxel, we estimated the magnitude of the responses in the single-step task, {gamma}sstepresp, and the posterior probability that the responses were greater than zero given the data, P{{gamma}sstepresp > 0|Y}, where "resp" denotes response magnitude and "sstep" refers to the single-step task.

We observed robust activity during the single-step task in each visual area. The majority (60%) of the hV4 voxels that exhibited a visual response also exhibited a response in the single-step task that reached a q ≥ 0.95 posterior probability threshold (Fig. 5A, dark gray bars). A substantial proportion of visually responsive voxels in V3A (43%) and V3 (35%) exhibited responses in the single-step task. We observed responses in the single-step task in only a about a quarter of visually responsive voxels in V2 (26%) and V1 (22%).


Figure 5
View larger version (17K):
[in this window]
[in a new window]

 
FIG. 5. Population activity in 3 task conditions. A: proportion of visually responsive voxels across all hemispheres in which responses reached a posterior probability threshold of q ≥ 0.95. Light gray bars, responses to ipsilateral visual stimuli during stimulus-only fixation task (stim). Medium gray bars, responses to saccades in the absence of salient visual stimuli (sac). Dark gray bars, responses in the single-step task when a visual stimulus appears in the ipsilateral visual field and is followed by an ipsiversive saccade (sstep). The prevalence of voxels activated by the single-step task increases with position in the visual hierarchy. B: response magnitude in three conditions normalized by responses to contralateral visual stimuli. Magnitude estimates were averaged across all visually-responsive voxels within a region of interest (ROI) within a single hemisphere. Box-and-whisker plots indicate the distribution of response magnitudes across hemispheres. The strength of activity in the single-step task increases with position in the visual hierarchy.

 
We next asked whether the magnitude of responses in the single-step task varies across visual area. We normalized the response in single-step trials in each hemisphere by the magnitude of the visual response, as described in METHODS. We found that the strength of responses in the single-step task increased monotonically as a function of position within the visual hierarchy (Fig. 5B, dark gray bars). The largest normalized responses were observed in areas hV4 (median value of 71%) and V3A (61%). Smaller responses were observed in areas V3 (35%), V2 (23%), and V1 (17%).

This monotonic relationship between response strength and position in the visual hierarchy is demonstrated by a series of pairwise comparisons. These comparisons were computed using Monte Carlo simulations (see METHODS). The largest response was observed in hV4. There was a high posterior probability (q ≥ 0.95) that the response in hV4 was larger than responses in V1–V3, and there was a high posterior probability (q = 0.64) that the response in hV4 is larger than the response in V3A. The next largest response was observed in V3A. There was a high posterior probability that this response is larger than responses in V1–V3 (q ≥ 0.91). The third largest response was observed in V3. There was high posterior probability (q ≥ 0.95) that this response is larger than responses in V1 and V2. Finally, the response in V2 was only marginally larger than the response in V1 (q = 0.63). This series of comparisons indicates that the strength of responses in the single-step task increases at each successive stage in the hierarchy.

These results indicate that the single-step task activates higher-order visual areas more strongly than early visual areas. This difference was evident in both number of activated voxels (prevalence) and the relative response strength (magnitude) across areas. As will be described in the following text, responses in the single-step task reflect activity due to the stimulus and to saccades, as well as to remapping. In subsequent sections, we perform several additional analysis aimed at isolating activity associated with remapping.

Responses in control conditions

As illustrated in the single-voxel example, both ipsilateral stimuli and saccades evoke small responses in visual cortex (Fig. 4, C and D). It is therefore possible that a portion of activity in the single-step task could be attributed to either the ipsilateral stimulus or to saccades alone rather than to remapping activity per se. In this section, we analyze activity in the two control conditions to determine the degree to which ipsilateral stimuli and saccades contributed to activity in the single-step task.

RESPONSES TO IPSILATERAL VISUAL STIMULI ALONE. In the single-step task, a visual stimulus flashes in the ipsilateral visual field. Although receptive fields in striate and extrastriate cortex are predominantly contralateral, it is conceivable that the ipsilateral stimulus itself elicited a response. This consideration is particularly important in areas hV4 and V3A because receptive fields increase in size at later stages of the visual system and some extend into the ipsilateral visual field (Gattass et al. 1981Go, 1988Go; Kastner et al. 2001Go). In the single-step task, it is possible that neurons with large receptive fields that extended into the ipsilateral visual field could have been driven by the stimulus to a greater degree than V1 neurons that have smaller receptive fields.

We assessed this possibility by measuring responses in a stimulus-only control condition (Fig. 1B). In this condition, subjects maintained fixation while a stimulus flickered in the ipsilateral visual field. This condition was balanced with the single-step task for orbital position and visual stimulation. The only difference between the two conditions was the presence or absence of the auditory cue and the resultant saccade. Fewer than 10% of visually responsive voxels in V1 and V2 responded to the ipsilateral stimulus with a posterior probability that reached a q ≥ 0.95 threshold (Fig. 5A, light gray bars). This observation indicates that the ipsilateral visual stimulus did activate a small proportion of voxels in each visual area. Ipsilateral responses were slightly more prevalent in areas V3, V3A, and hV4, with between 10 and 13% of voxels reaching threshold in each area. Although these responses in the stimulus-only condition reveal that some voxels in each area are activated by the ipsilateral stimulus, such responses do not account for the activation we observed in the single-step task. Ipsilateral responses were far less prevalent in the stimulus-only condition than in the single-step task.

We considered whether there were differences across visual areas in the magnitude of ipsilateral responses (Fig. 5B, light gray bars). Areas V3A and hV4 had the largest ipsilateral response (median values of 7 and 5% of the contralateral visual response). The posterior probabilities that either of these responses were larger than responses in any of the other cortical areas were slightly greater than chance (0.50 < q < 0.75). These comparisons indicate a small increase in the magnitude of ipsilateral responses in later stages of the hierarchy. Even the largest responses, however, were small relative to visual responses in these same voxels. We conclude that ipsilateral responses are both too weak and too sparse to account for the relatively large responses observed in the single-step task.

RESPONSES TO SACCADES ALONE. A potential concern is whether saccades alone activate visual cortex. This issue is particularly important in higher-order visual areas. Neurons in both V4 and V3A fire in relation to saccades directed toward their visual receptive fields (Nakamura and Colby 2000Go; Tolias et al. 2001Go). Furthermore, differences between areas in receptive field size could have increased the chances of observing saccade-related activity in V3A and V4 relative to other visual areas. The logic is as follows. In the single-step task, subjects made 16° saccades. Thus visual responses associated with processing the saccade target were located in the 16° representation in the cortical retinotopic map. Because receptive fields are larger in V3A and hV4, the 16° representation is more likely to overlap with the expected site of remapped activation (9°) in V3A and hV4 than in area V1. It was therefore critical that we determine the extent to which saccades contribute to responses in the single-step task.

We addressed this issue by testing subjects on a saccade-only control condition (Fig. 1C). This condition was balanced with the single-step task for orbital position, auditory stimulation, and number of saccades. The only difference between the two conditions was the presence or absence of the visual stimulus in the 1 s preceding the cue to initiate a saccade. We found that saccades in the absence of the visual stimulus did activate voxels in each visual area (Fig. 5B, medium gray bars). A minority of visually responsive voxels in V1 and V2 (12 and 14%) responded in the saccade-only condition with a posterior probability that reached a q ≥ 0.95 threshold. Saccade-related responses were more prevalent in areas V3 (20%), V3A (24%), and hV4 (22%) than in V1 and V2. This analysis indicates that saccades may have contributed to activity in the single-step task. Moreover, the contribution of saccade-related activity was larger than the contribution of ipsilateral visual responses. However, responses in the saccade-only condition were still less prevalent than in the single-step task (Fig. 5A, medium vs. dark gray bars), indicating that responses in the single-step task cannot be attributed primarily to saccades.

We considered whether there were differences in the magnitude of saccade-related responses across visual areas (Fig. 5B, medium gray bars). The largest responses in the saccade-only control condition were observed in areas V3A and hV4 (median values of 24 and 16% of the visual response, respectively). The responses in V3A and hV4 were not statistically different (q = 0.53). Neither V3A nor hV4 had a high probability of being larger than responses in V3 (q < 0.80). However, there was a high probability (q ≥ 0.95) that responses in V3, V3A, and hV4 were all larger than responses in V1 and V2. Finally, responses in V1 and V2 were not different from each other (q < 0.5). These pairwise comparisons indicate that saccades activate high-order visual areas to a greater extent than V1 and V2.

In conclusion, the analysis of responses in the saccade-only control condition revealed that saccades in the absence of a salient stimulus do elicit responses. Furthermore, the pattern of activity is similar to that observed in the single-step task in that responses are strongest in higher-order visual areas. Across all visual areas, saccade-related responses were smaller than responses in the single-step task, indicating that saccades alone do not account for remapping activity measured in the single-step task.

Responses in single-step task are larger than the sum of control responses

Analysis of responses in the control conditions indicates that both ipsilateral stimuli and saccades activate striate and extrastriate visual areas to some degree. Activation in the single-step task potentially reflects three factors—visual, saccade, and remapping. We asked whether responses in the single-step task could be accounted for by the simple linear summation of responses to ipsilateral visual stimuli and saccades as measured in the two control conditions. There are two possible outcomes to this analysis. If responses in the single-step task equal the summed activity in the two control conditions, we can conclude that the presence of remapping is small or nonexistent. If, on the other hand, responses in the single-step task exceed the summed activity in the two control conditions, we would conclude that remapping is present in early visual cortex, despite the presence of nontrivial activity in the control conditions.

Selectivity indices provide a convenient method for visualizing the relative strength of responses evoked by sets of conditions. We calculated a three-way selectivity index, S, as follows. For each of the three conditions, c, we calculated

Formula 2(2)
where Formula 2c is the proportional signal change for a given condition averaged over all visually responsive voxels in a given cortical area. Here, the subscript, i, denotes the ipsilateral version of each of the three task conditions. Selectivity values, Sc, sum to 1 Formula 2 and are all nonnegative.

The results from this three-way selectivity index are represented on triangular simplex plots (Fig. 6). Position in the simplex was determined as follows. Let Vsstep = (0, {surd}3/2), Vmac = (–,0), Vstim = (,0) be the vertices of a triangle. The plotted positions of a hemisphere in the simplex are given by Formula 2 Position in the simplex represents the degree to which the MR response is selective for each of the three conditions. For example, a voxel that responds most strongly on single-step trials relative to the two control conditions will be located in the top sector. A voxel that responds equally strongly in all three task conditions will be represented in the middle of the simplex.


Figure 6
View larger version (27K):
[in this window]
[in a new window]

 
FIG. 6. Responses in the single-step task are larger than responses in both control conditions. Each dot represents the average activity in all 3 conditions for a single hemisphere: ipsilateral responses in the single-step task, stimulus-only control condition, and saccade-only control condition were each averaged within hemisphere, normalized so that they sum to 1, and then plotted on triangular simplex plots. Position of a hemisphere in the simplex represents the selectivity of responses for each of the three conditions. Dotted horizontal line indicates the position in the triangle at which the sum of the 2 control responses—saccade-alone and stimulus-alone—equals the response in the single-step task. Grayscale shading corresponds to the posterior probability that a particular hemisphere fell above this summation line (see Eq. 3).

 
For each visual area, indices from the majority of hemispheres fell within the top sector of the triangular simplex plots (Fig. 6). This indicates that performance of the single-step task elicited larger responses than either of the two control conditions. In the single-step task, an ipsilateral visual stimulus appears and subjects make a saccade. The critical analysis therefore compares the magnitude of the response in the single-step task to the sum of responses to ipsilateral stimuli and saccades. The dotted horizontal line in each simplex plot indicates the location in the simplex that results from the linear summation of activation in the two control conditions. We refer to this line as the summation line. The majority of indices from hemispheres in each visual area fell above the summation line, indicating that responses in the single-step task cannot be attributed to the sum of visual and saccade-related activity.

The position of each point in the simplex is determined by the mean of the estimated response. There is a degree of uncertainty regarding each position that is reflected by the posterior SD of the estimate. For example, responses from a given hemisphere would be located in the top sector of the simplex plot if the mean of the posterior distribution in the single-step task was large relative to the mean in the two control conditions. However, there would be low certainty regarding the position of that hemisphere in the simplex if the SD of each distribution was also large. We calculated the following probability that takes this uncertainty into account. From the distribution of the response parameters, {gamma}remap, {gamma}sac, and {gamma}stim, under the Bayesian model, we derived the posterior probability that responses fell above the summation line

Formula 3(3)
High posterior probability values (q ≥ 0.95) indicate strong evidence for remapping. Probability values near chance (q = 0.5) indicate that remapping and linear summation cannot be distinguished.

A substantial proportion of voxels exhibited responses in the single-step task that were larger than the sum of responses in the two control conditions. The Bayesian analysis revealed that 43% of voxels in area hV4 had a high posterior probability (q ≥ 0.95) of exhibiting a stronger responses in the single-step task in than in both control conditions combined. About one-fifth of voxels in V3 (21%) and V3A (20%) and fewer than one-fifth of voxels in V1 (15%) and V2 (14%) reached this same threshold. The analysis of linear summation indicates that remapping is considerably less prevalent in early visual areas than in area hV4. This analysis of linear summation provides strong evidence for the existence of remapping in area hV4. For reasons that will be discussed in the following text, this analysis may be overly conservative.

Remapping and the subadditivity of hemodynamic responses

In many experimental contexts, it is reasonable to assume that the hemodynamic response elicited by two neural events is equal to the sum of responses measured independently—the hemodynamic response function approximates a shift-invariant linear system (Boynton et al. 1996Go). However, there is a growing consensus that hemodynamic responses behave nonlinearly in specific contexts (Birn and Bandettini 2005Go; Birn et al. 2001Go; Friston et al. 1998Go, 2000Go; Huettel and McCarthy 2001Go; Vazquez and Noll 1998Go). For example, many studies have shown that there is a saturating nonlinearity for closely spaced neural events (review in Wager et al. 2005Go). Two visual stimuli that occur in rapid succession evoke an MR response that is smaller than would be predicted by the sum of responses to the two stimuli in isolation. In other words, closely spaced events can sum sublinearly. This effect can result in as much as a 50% decrease i.e., It is not yet known whether sublinear summation of responses reflects true nonlinearities in the hemodynamic response function, nonlinearities in the stimulus-induced neural response, or a combination of the two. A saturating nonlinearity could have affected responses in the single-step task. In this task, two events—a stimulus and a saccade—occur in rapid temporal succession. The resultant response may thus be smaller than would be predicted by the sum of responses to the stimulus and saccade measured in isolation. This nonlinearity would cause the analysis of linear summation in the previous section to be overly conservative.

The issue of hemodynamic nonlinearity is difficult to address directly. The neural phenomenon of remapping is itself nonlinear—the conjunction of a stimulus and a saccade produces a response that is not there if either occur alone. Our approach to estimating the sublinear summation present in our data was to analyze responses in three new conditions. Specifically, we analyzed responses in the single-step task when the stimulus appeared in the contralateral visual field and was followed by a contraversive saccade, the stimulus-only fixation task when the stimulus appeared in the contralateral visual field, and the saccade-only condition in which the saccades matched the saccades on single-step trials. These three conditions mirrored the three experimental conditions in that the stimulus was located in the contralateral rather than the ipsilateral visual field, and the saccade was contraversive rather than ipsiversive.

The logic behind this analysis is as follows. By definition, we did not expect remapping in the contralateral hemisphere. If responses sum linearly, contralateral responses in the single-step task should simply reflect the sum of responses in the stimulus-only and saccade-only conditions

Formula 4(4)
where "c" denotes the contralateral hemisphere. In contrast, we expect that activity in the ipsilateral hemisphere in the single-step task reflects responses associated with each of the two control conditions and responses associated with remapping

Formula 5(5)
where "i" denotes the ipsilateral hemisphere. In the linear case, we get a reasonable estimate of Formula 5 by subtracting the estimates Formula 5 and Formula 5 from Formula 5. But if instead the responses combine subadditively, this will underestimate the remapping contribution. To account for this potential subadditivity, we use the contralateral responses as a control to estimate the degree of subadditivity. Although the combination of closely spaced responses will in general be linear, we approximate the subadditivity by a linear shrinkage of the sum. That is, we write sstepc = u(saccc + stimc), where 0 ≤ u ≤ 1. (Constraining u to the interval [0,1] precludes the possibility of superadditive summation.) We thus estimate the parameter u by

Formula 6(6)
where we require the estimate to lie in [0,1] as well. This û represents the degree to which responses in the contralateral hemisphere are smaller than would be predicted by linear summation. Values of 0 indicate strong sublinear summation; values of 1 indicate that responses in the contralateral hemisphere sum linearly. Given the value û, we estimate the remapping contribution as what is left over in Formula 6 after subtracting the corrected combination of Formula 6 and Formula 6

Formula 7(7)
Here, we ignore any subaddivity in the combination of the three effects, which to first order should make Formula 7 a conservative estimate of the remapping contribution.

In a final analysis stage, we used the Bayesian model to calculate the posterior probability that Formula 7 is larger than 0, conditional on the subadditivity parameter

Formula 8(8)

We quantified the degree of sublinear summation in the contralateral hemisphere by calculating the parameter û. As described in Eq. 6, small values of û indicate large subadditivity. The majority of voxels in each visual area had û values that were <1, indicating that contralateral responses in the single-step task were smaller than predicted by linear summation of activity in the control conditions (Fig. 7A). We found that û was smallest in area V1 (median û = 0.76), and was largest in hV4 (median û = 0.96). This is notable because hemodynamic nonlinearity has been studied most extensively in primary visual cortex. Our results suggest that the degree of nonlinearity is variable across voxels and across with cortical areas.


Figure 7
View larger version (27K):
[in this window]
[in a new window]

 
FIG. 7. A: distribution of subadditivity parameter (û) calculated from Eq. 6. Values of 1 indicate that contralateral responses in the single-step task reflect the linear sum of activity in the stimulus-only and saccade-only control conditions. Values <1 indicate that responses in the contralateral hemisphere are smaller than would be predicted by linear summation. Vertical line in each plot indicates the median of the distribution. B: proportion of voxels for which the posterior probability that Formula 15 reached a q ≥ 0.95 threshold. C: average Formula 15 for each hemisphere. The calculation of Formula 15 takes into account activity in both control conditions and estimated subadditivity of responses. Each dot represents the averaged response from a single hemisphere. Grayscale shading corresponds to the posterior probability that Formula 15 was >0, given the subadditivity parameter, û.

 
We estimated the magnitude of remapping, Formula 8, in the ipsilateral hemisphere, according to Eq. 8. This estimate reflects the relative magnitudes of responses in all six trial types (3 conditions, 2 directions). A substantial proportion of voxels in each visual area had nonzero Formula 8 values (Fig. 7B). This estimate of remapping strength is less conservative than the simple linear analysis because the scaling factor, û, reduces the size of the summed visual and saccade activity. Finally, we used the Bayesian model to calculate the probability that Formula 8 was larger than zero, conditional on û (Fig. 7C).

The analysis of sublinear summation provides three important insights. First, it indicates that responses did sum sublinearly and that nonlinearities are important to consider when measuring responses to rapidly occurring events. Second, this analysis indicates that remapping is present throughout occipital cortex. Third, this analysis reveals a monotonic relationship between the magnitude of remapping and position in the visual hierarchy.

Time course of visual and remapped responses

Remapping occurs at various points in time relative to saccade initiation. A substantial proportion of neurons in parietal and extrastriate cortex remap predictively, while others begin to respond around the time of the saccade (Duhamel et al. 1992aGo; Nakamura and Colby 2002Go). Predictive responses occur at a latency that is shorter than the typical visual response for that neuron. A subset of cells with predictive responses begin to respond to the stimulus trace even before the eyes have moved.

This neural variability in the timing of remapping relative to the saccade should not be observable in our fMRI data. The hemodynamic response function acts as a low-pass filter of the underlying neural activity, obscuring small temporal variations in response onset time. We did expect to observe a difference in response time between visual and remapped activity. In the fMRI version of the single-step task, the stimulus appears and stays on the screen for 1 s prior to the auditory cue to make an eye movement. We expect that remapping occurs around the time of the eye movement. Subjects in this study had a mean saccadic reaction time of 255 ± 128 ms. Thus the onset of the stimulus preceded the onset of the saccade by an average of 1,255 ms, relative to the start of the trial. Because of this interval, remapped responses driven by the stimulus trace should begin ~1,255 ms later than visual responses driven by the stimulus. This statement assumes that remapping occurs around the time of the eye movement.

We used the Bayesian estimates of response profile to test this prediction. The three relevant model parameters are lag, attack, and decay. The first two parameters—lag and attack—correspond to early stages of the response. Lag corresponds to the time from stimulus onset to the start of the hemodynamic response; lag is equivalent to response latency. Attack corresponds to the time from the start of the response to the peak of the response; attack is a measure of the rate at which the response rises. Finally, the decay parameter corresponds to the duration from the peak of the response to the point that the response returns to baseline levels; decay measures the duration of response offset.

We predicted that remapped responses would have a longer lag and attack than visual responses but that the two response types would not differ in decay. Our analysis of response profile was performed on the subset of voxels in each visual area in which both visual and remapped responses had a high probability of being nonzero (q ≥ 0.95). This selection criterion was necessary because inferences regarding response profile can only be based on actual responses. The analysis described in this section includes a smaller proportion of visually responsive voxels that the analyses described in previous sections because, as we showed in Fig. 5, only a subset of visually responsive voxels in each visual area responded to the stimulus trace.

ANALYSIS OF TIME-TO-PEAK (LAG+ATTACK). We calculated response peak by simply summing the lag and attack parameters ({gamma}lag + {gamma}attack). Combining the lag and attack parameters gives a measure of the time-to-peak relative to the onset of the stimulus. We compared estimates of time-to-peak for visual responses and remapped responses. Our goal was to determine if the difference in time-to