| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
1Department of Neuroscience, University of Pittsburgh; 2Department of Statistics, Carnegie Mellon University; and 3Center for the Neural Basis of Cognition, Pittsburgh, Pennsylvania
Submitted 21 February 2006; accepted in final form 1 November 2006
 |
ABSTRACT |
|---|
 TOP ABSTRACT INTRODUCTIONMETHODSRESULTSDISCUSSIONAPPENDIXGRANTSACKNOWLEDGMENTSREFERENCES |
|---|
|
INTRODUCTION |
|---|
|
TOPABSTRACTINTRODUCTIONMETHODSRESULTSDISCUSSIONAPPENDIXGRANTSACKNOWLEDGMENTSREFERENCES |
|---|
; Goldberg et al. 1990; Gottlieb et al. 1998; Heiser et al. 2005; Kusunoki et al. 2000). In these studies, the monkey fixates and a stimulus is briefly presented outside the receptive field of the neuron being recorded. When a new fixation point appears, the monkey makes an eye movement to it. This eye movement brings the receptive field onto the recently stimulated screen location. LIP neurons become active at this time even though there is no physical stimulus in the receptive field. This activity is interpreted as a response to the memory trace of the brief visual stimulus. Updating depends on a corollary discharge of the eye-movement command signal (Sommer and Wurtz 2002). The response to the updated trace of the stimulus indicates that LIP neurons perform a coordinate transformation that enables the visual system to maintain perceptual continuity across saccades (Colby and Goldberg 1999).
Parietal cortex is centrally important in creating an updated representation of space. The majority of LIP neurons exhibit remapping (Duhamel et al. 1992a), and reversible inactivation of LIP impairs performance on tasks that require updated spatial information (Li and Andersen 2001). Studies in humans also indicate a central role for parietal cortex. Neurological patients with parietal lobe lesions are impaired on tasks that require spatial updating (Duhamel et al. 1992b; Heide et al. 1995; Khan et al. 2005). Imaging studies have also demonstrated the role of parietal cortex in remapping. We have used fMRI to show that representations of visual stimuli are updated from one hemisphere to the other in conjunction with horizontal single-step saccades (Merriam et al. 2003). Similarly, several fMRI studies have used double- and triple-step saccade tasks to demonstrate remapping in humans (Heide et al. 2001; Medendorp et al. 2003, 2005a,b). These studies indicate that there is a functional similarity between the computations performed by parietal neurons in monkeys and humans (Crawford et al. 2004; Merriam and Colby 2005).
Remapping activity is not limited to parietal cortex. Remapping has been observed in the frontal eye field (FEF), the superior colliculus (SC), and extrastriate visual cortex. Neurons in all these areas have spatially selective visual and perisaccadic responses, are modulated by spatial attention, and respond to the stimulus trace in the single-step saccade task (Nakamura and Colby 2002; Umeno and Goldberg 1997, 2001; Walker et al. 1995). If remapping is important for perceptual constancy, remapping should not be limited to brain regions with attentional and oculomotor functions. Rather, updated spatial information should reach visual areas that are involved in visual perception. The goal of the present study was to test the hypothesis that updating occurs in early visual cortex in humans. Two lines of evidence suggests that it does. First, psychophysical studies have demonstrated that updated visual signals are required to integrate information about stimulus features across saccades (Hayhoe et al. 1991; Melcher and Morrone 2003; Prime et al. 2006). Second, several human fMRI studies have demonstrated strong top-down effects throughout occipital cortex. Multiple visual areas are activated in tasks that involve spatial attention (Brefczynski and DeYoe 1999; Gandhi et al. 1999; Kastner et al. 1999; McMains and Somers 2004; Ress et al. 2000; Silver et al. 2005; Tootell et al. 1998; Yantis et al. 2002). Many of these areas are also modulated by oculomotor signals (DeSouza et al. 2002; Sylvester et al. 2005). These fMRI studies indicate that visual cortex has access to the corollary discharge signals necessary for remapping.
We used an fMRI version of the single-step task to test whether remapped visual signals are present in visual cortex. In this task, subjects fixate while a salient visual stimulus flickers in the periphery. The stimulus is expected to activate visually responsive cortex in the contralateral hemisphere. The stimulus is then extinguished, and a tone cues the subject to make an eye movement to a stable target. The target position is chosen so that the eye movement brings the location of the now-extinguished stimulus into the opposite hemifield. The premise of this experiment is that activation related to the memory trace of the stimulus is remapped from one hemisphere to the other with the eye movement. We predicted that the hemisphere that was initially ipsilateral to the stimulus would become active around the time of the eye movement. We found strong evidence for remapping in striate cortex and in each extrastriate visual area examined. Further, we found that remapping was more robust in higher-order extrastriate areas. Our results indicate that remapping is present in visual areas that are directly linked to visual perception.
|
METHODS |
|---|
|
TOPABSTRACTINTRODUCTIONMETHODSRESULTSDISCUSSIONAPPENDIXGRANTSACKNOWLEDGMENTSREFERENCES |
|---|
We studied a total of 14 healthy participants (7 female, aged 25–35). All subjects had extensive prior experience with both fMRI and psychophysical experiments. Informed written consent was obtained in accordance with the University of Pittsburgh IRB. All subjects had normal or corrected vision. Data from two subjects were discarded because of noise artifacts in the MR data.
Behavioral paradigms
Visual stimuli were generated on a PC computer using the Psychophysics Toolbox (Pelli 1997) running in Matlab 6.5 (Mathworks, Natick, MA). Stimuli were presented via an LCD projector and long-throw optics onto a back-projection screen in the bore of the MR scanner. Subjects viewed the projected stimuli through an angled mirror, resulting in a 10° vertical x 20° horizontal field of view. We measured fMRI activation while subjects performed three tasks, as described in the following text.
SINGLE-STEP TASK. Two stable crosses were located 8° to the left and right of screen center. Subjects fixated one of the two crosses at the beginning of the trial (Fig. 1A). After a variable fixation period (1,000 ± 200 ms), a small (1–2°) visual stimulus appeared at the center of the screen and 3° above the horizontal axis in the upper quadrant of the right or left visual field. Subjects were instructed to maintain fixation on the cross and not to look at the stimulus. After 1 s, the stimulus was extinguished, and a tone cued the subject to make an eye movement to the opposite fixation cross. This saccade brought the screen location of the now-extinguished stimulus into the opposite visual field. The trial ended after a variable period of fixation (2,000 ± 200 ms) when a second tone instructed the subject to make a return saccade back to the initial fixation cross.
|
SACCADE-ONLY TASK. This task is similar to the single-step task. The only difference is that no salient visual stimulus appears prior to the eye movement (Fig. 1C). Two crosses were located 8° to the left and right of screen center. Subjects fixated one of two crosses at the beginning of the trial. After a variable period (2,000 ± 200 ms), a tone cued the subject to make an eye movement to the opposite fixation cross. Subjects maintained fixation for a variable period (2,000 ± 200 ms) until a second tone instructed them to make a return saccade back to the initial fixation cross. The timing of the task was identical to the single-step saccade task.
The stimulus-only fixation task and saccade-only task were intended to control for sensory and motor factors of the single-step task that are not specific to remapping. For example, saccades in both the single-step task and the saccade-only task were triggered by an auditory cue. Activation attributable to the auditory stimulus should thus be equal in the two tasks.
Trials of each condition lasted an average of 4,000 ms, including the variable fixation period. We found that varying the length of the initial fixation period from trial to trial reduced the number of instances in which subjects made an eye movement prior to the auditory cue. As a consequence of this variability in fixation duration, the start time of each trial was not yoked to the scanner TR. We recorded trial timing information, eye position, and scanner pulses using custom software. We then used this timing information in the analysis of the fMRI data.
Experimental design
Each subject participated in at least two scanning sessions, one for the main remapping experiment, and another session for identifying the location of each visual area using retinotopic mapping procedures (described in the following text). A single scanning session lasted 
1.5 h; each session consisted of 6–10 runs; each run lasted 512 s; 64 trials were tested in each run. On a subset of runs (3–5), all trials began with fixation on the left cross, and on the other subset of runs, trials began with fixation on the right cross. Each of the three tasks was performed in both directions over the course of the session. This was a critical feature of the experimental design because it enabled us to measure responses in each hemisphere when the stimulus was located in either the contralateral or ipsilateral visual field. The two initial fixation positions were never mixed within a run: on runs in which the stimulus appeared in the left visual field, the stimulus never appeared in the right visual field. During the course of a scanning session, subjects performed 128–256 repetitions of each task.
We used an experimental design in which periods of fixation were interspersed with experimental trials. During the periods of fixation, or null trials, subjects simply maintained fixation on the initial cross for 4,000 ms. Periods of fixation were matched with experimental trials for orbital position, duration, and frequency. The ordering of experimental and null trials was determined by a special class of pseudorandom sequences known as m-sequences (Reid et al. 1997; Sutter 2001). Randomly generated stimulus sequences often have temporal autocorrelations that can interfere with response estimation. In contrast, m-sequences have a nearly-flat autocorrelation function. This property makes m-sequences advantageous for fMRI (Buracas and Boynton 2002; Liu 2004; Liu and Frank 2004).
Eye-position recording
We monitored eye position during each fMRI session using a video-based eye tracker (ASL, Boston, MA). The eye tracker had a temporal resolution of 60 Hz. Stability of eye data was typically better than 2° as determined by the SD of the data during periods of stable fixation. Figure 2, A–C, shows eye traces from a single subject recorded during scanning.
|
We used an automatic saccade finding algorithm to search for saccades in the first two seconds of each trial [http://groups.yahoo.com/group/ilab (Gitelman 2002)]. The software identified the occurrence of a saccade if eye velocity exceeded 50 °/s and the eyes moved >2°. Subjects were expected to have made saccades in response to the auditory cue on single-step and saccade-only trials. Mean saccadic reaction time was 255 ± 128 (SD) ms in the single-step task and 269 ± 136 ms in the saccade-only condition. The difference in saccadic reaction time was not significant (t-test, P > 0.05). It was critical to our experiment that subjects not make a saccade while the stimulus was still present on the screen. On occasional trials, subjects made an early saccade, while the stimulus was still present and prior to the auditory cue. These early-saccade trials were not included in the analysis of the fMRI data.
MRI data acquisition and preprocessing
We used functional magnetic resonance imaging at 3T (Allegra, Siemens, Erlangen) and a T2*-sensitive EPI pulse sequence to measure changes in BOLD activity. Scan parameters were as follows: TR = 1,000 ms; TE = 30 ms; flip angle = 65°. We collected 16 slices (3 mm3 voxels, 192 mm field of view) in each volume, and 512 volumes in each functional run. Slices were oriented perpendicular to the calcarine sulcus to cover the entire occipital lobe (Fig. 3A). Functional data were preprocessed using FIASCO software [http://stat.cmu.edu/ fiasco (McNamee and Eddy 2001)]. Preprocessing steps included correction for fluctuations in mean intensity; motion correction of the raw k-space data (Eddy et al. 1996); image reconstruction, and outlier correction using a Windsor filter. Outliers were defined as data points farther than 10 times the interquartile range from the median. The reconstructed MR images were not smoothed, temporally filtered, or spatially normalized.
|
We used a fully Bayesian approach to analyzing the MR data, the details of which have been described elsewhere (Genovese 1998a,b, 2000) (see APPENDIX). Briefly, we fit the fMRI time series data with a nonlinear, hierarchical model that decomposes the observed signal into four components: baseline, drift, activation, and noise. The activation component is further subdivided into lag, attack, and decay. We used Bayesian statistical methods to derive the posterior distribution of the parameters given the data, P(
|Y). From this distribution, we computed posterior probabilities related to our questions of interest as well as point estimates (posterior means) and their posterior SDs. All inferences in this study are based on these posterior distributions of the parameter vector through derived posterior quantities. This analysis yields a Bayesian posterior probability, which we denote as "q." The posterior probability should not be confused with a P value from a classical statistical test (i.e., the probability that the data could be drawn from the population tested given the assumption that the null hypothesis is true).
We make four key inferences in this study. The first is the probability that there is a nonzero response in a task condition. For condition "c," this is denoted by p{cresp > 0|Y}, where cresp is the response magnitude parameter of the model. Because our hierarchical model allows for a nonzero probability on the discrete value 0, this probability indicates the strength of evidence for a response above baseline. The second inference that we consider is the probability that the response magnitude in one condition, c, is greater than the response magnitude in another, c', denoted by P{cresp > c'resp|Y}. We make similar comparisons for shape parameters, such as when comparing response onset times across conditions, denoted by P{cttp > c'ttp|Y}. Third, we make inferences regarding more complex events, such as the posterior probability that the response in one condition, c, is larger than the combined responses in two other conditions, c' and c'', denoted by P{cresp > (c'resp + c''resp)|Y}. Finally, we make inferences about the population by combining the voxel-wise point estimates across voxels in a visual area in a single hemisphere, and across hemispheres and subjects, yielding p(
|Y). This enables us to make group-level comparisons across visual areas. For example, we compare the combined size of a response in a given cortical area, a, to that in a another area, a', denoted by P{aresp > a'resp|Y}.
Hierarchical Bayesian statistical models confer numerous advantages over models that use classical statistical procedures (for discussion, see Genovese 2000). The use of Bayesian statistics has become increasingly common in fMRI data analysis and a number of Bayesian statistical models have been described in the literature (see Friston and Penny 2003; Friston et al. 2002a,b; Genovese 2000; Gössl et al. 2001; Marrelec et al. 2003, 2004; Penny et al. 2003, 2005; Smith et al. 2003; Woolrich et al. 2004a,b).
Voxel selection
We used three criteria to select functional voxels. First, we identified gray-matter voxels. Second, we selected voxels based on inclusion within the boundaries of predefined cortical visual areas and omitted voxels that straddled borders. Third, we selected the subset of voxels that responded to the small and brief contralateral stimulus in the stimulus-only fixation task. Each of these criteria are described in more detail in the following text.
GRAY-MATTER SEGMENTATION.
We acquired two or three three-dimensional (3D) anatomical volumes from each subject using an MPRAGE pulse sequence (30 ms TE, 8° flip angle, 192 slices, 1 mm3 voxel resolution) and averaged the volumes to increase the signal to noise ratio. Whole-brain 3D anatomical images for each subject were registered to that subject's functional data using a fully automated algorithm implemented in FIASCO software. Gray matter, white matter, and CSF were then segmented using FreeSurfer software http://surfer.nmr.mgh.harvard.edu (Fischl et al. 1999); http://surfer.nmr.mgh.harvard.edu (Dale et al. 1999). We used the estimated boundary between white and gray matter to select functional voxels for further analysis. We visualized activation on 2D flattened representations of the cortical surface (Fig. 3, C–F). Flat maps were created using the mrVista MATLAB toolbox [http://white.stanford.edu/software (Wandell et al. 2000)].
RETINOTOPIC SPECIFICITY.
We used standard retinotopic mapping procedures to identify the borders between occipital lobe visual areas V1, V2, V3, V3A, and hV4 (DeYoe et al. 1996; Engel et al. 1994, 1997; Sereno et al. 1995). Retinotopic mapping was carried out in a separate scan session. Subjects underwent six to eight runs of eccentricity and polar angle mapping (Fig. 3, C and D). We used retinotopic mapping stimuli that moved smoothly across visual space, creating a traveling waves of cortical activity. Stimuli were contrast and hue-modulated flickering checkerboards that took the shape of rotating wedges and expanding/contracting annuli. The details of the stimuli have been reported elsewhere (Tootell et al. 1997). The spatial frequency of the checkerboards was scaled to accommodate larger receptive fields in the periphery. Stimulus movement was periodic, with a frequency of 1/64 s. The stimulus completed 8.5 cycles/run. We used the phase cancellation technique described by Kalatsky and Stryker in which the direction of stimulus movement was reversed on successive runs. We summed the complex-valued data prior to calculating the phase and magnitude of the response. This procedure removes the hemodynamic delay associated with the blood-level-oxygen-dependent (BOLD) response, thereby yielding more accurate estimates of the stimulus position that elicited the maximal response.
The retinotopic mapping stimuli encompassed a large portion of the visual field (20° horizontal x 15° vertical) and hence activated a broad region of cortex (Fig. 3, C and D). Visual area boundaries were defined using a conjunction of polar angle and eccentricity maps according to the following three criteria, as described by Dougherty et al. (2003). First, each area was bounded by phase reversals in the angular component of the retinotopic map. Second, a given area had to be activated by both the wedge and annulus stimuli. Third, the phase gradient in the angular and eccentricity maps had to run orthogonal to one another. Both the dorsal and ventral portions of areas V1, V2, and V3 were easily identified using these criteria.
We identified area hV4 using the criteria described by Tootell and Hadjikhani (2001). Area hV4 is the ventral retinotopic area that continues laterally from ventral V3. Area hV4 has a full hemifield representation and is located proximal to the medial lip of the collateral sulcus. We refer to this area as "hV4" because the degree of functional correspondence between this region and monkey area V4 has not been fully resolved (Brewer et al. 2005; Tootell and Hadjikhani 2001). It is possible that the cortical area that we labeled hV4 also contains additional subdivisions (e.g., areas VO-1 and VO-2) (Brewer et al. 2005).
We identified area V3A using the criteria described by Tootell et al. 1997 Area V3A is the dorsal retinotopic area that continues anteriorly from dorsal V3. Area V3A contains a full hemifield representation and is located proximal to the transverse occipital sulcus at the base of the intraparietal sulcus. It is possible that the cortical area that we identified as V3A also contained other cortical areas (e.g., V3B) (Press et al. 2001). Area V7 was not reliably identified in our data and was therefore not included in this study.
Of the 24 hemispheres in this experiment, all but 2 had clean retinotopic maps in which the borders between visual areas could be determined unambiguously. In two hemispheres (1 left, 1 right) from two different subjects, the borders between visual areas were not clearly discernible. These hemispheres were not included in the analysis. Hence, all analyses report results from n = 22 hemispheres from 12 participants.
SELECTING VOXELS BASED ON VISUAL RESPONSIVENESS.
In the version of the single-step task used in neurophysiology experiments, an eye movement brings the receptive field of the neuron onto the recently stimulated screen location (Duhamel et al. 1992a). Remapped activity is observed as a response to the memory trace of the stimulus in the absence of any direct visual stimulation. Implicit in the logic of this paradigm is the assumption that the neuron fires when an actual stimulus appears inside of its receptive field. The goal of this third voxel selection procedure was to identify the set of voxels that respond to the visual stimulus used in this experiment—these are the same voxels that we predict could also exhibit remapping.
In both the single-step task and the stimulus-only fixation task, a small (1–2°) stimulus appears in the upper visual field for 1 s (Fig. 1, A and B). This stimulus should activate only a subset of the voxels that are activated by the large checkerboard stimuli in the retinotopic mapping experiment. We identified this subset of voxels by analyzing responses in the stimulus-only fixation task when the stimulus appeared in the contralateral visual field. For each voxel, we estimated the magnitude of the visually evoked response, visualresp, and the posterior probability that this magnitude was greater than zero given the data, P{visualresp > 0|Y}. Results from this analysis are plotted on the cortical surface (Fig. 3E). In this plot, activation magnitude is represented by a red–yellow color scale: voxels with large visual responses are yellow and voxels with smaller visual responses are red. Posterior probability is represented by color opacity with zero probability being fully transparent; no statistical threshold has been applied to these results. Yellow voxels tend to be more opaque because large responses tend to have a higher probability of being nonzero.
We measured visual responses in the stimulus-only fixation task when the stimulus appeared in the contralateral visual field. These visual responses had two notable properties (Fig. 3E). First, voxels activated by the visual stimulus were located in the appropriate region of each retinotopic map. In the fixation task, the stimulus appeared 3° above the horizontal axis and 8° from the vertical axis, in the upper quadrant of the right or left visual field. We thus expected the stimulus to activate the contralateral upper visual field representation at 9° eccentricity. This location in cortex is indicated by shades of magenta in the polar angle map (Fig. 3C) and by shades of yellow/green in the eccentricity map (Fig. 3D). These activation maps indicate that visual responses in the fixation task were located in the subregion of each visual area that correspond to the appropriate location in the retinotopic map.
Second, there is a clear distinction between active and inactive voxels. Posterior probabilities tended to be either high (far greater than chance, q 
0.5), or low (far less than chance, q << 0.5). Because of this property, voxels appear as either fully opaque or completely transparent (q approaches 0 in cortical locations in which the underlying grayscale anatomy is clearly visible). All subsequent analyses were performed on the subset of voxels in each area that met a q 
0.95 selection criteria for contralateral visual stimuli. The sharp distribution of probability values indicates that the particular threshold did not have a strong impact on which voxels were included in the analysis.
Response normalization
There was considerable variability across subjects and visual areas in the magnitude of the visual response in the stimulus-only fixation task. It is not clear whether this variability is due to differences in neural response strength or to interregional and intersubject differences in hemodynamics. Logothetis and Wandell (2004) have argued that regional differences in the coupling between neural activity and hemodynamic changes could result in spurious differences in response magnitude across cortical areas. They termed this coupling hemodynamic response efficiency or HRE. It is common to normalize MR activation by the baseline signal level on a voxelwise basis, thereby expressing activation in units of percent signal change. However, normalizing by the baseline does not account for regional differences in HRE. One solution to this problem is to normalize responses by the magnitude of activation in a second condition that is known to elicit a consistent response. Such selectivity measures reflect the proportional increase or decrease in activation in a particular area, given the regional HRE. The data in this experiment lend themselves to this normalization procedure because voxels were selected based on there being a highly probable visual response. Based on this line of reasoning, we normalized activation magnitudes in each condition by the magnitude of the visual responses. Normalized values were calculated as follows. We first identified the set of visually responsive gray matter voxels in a given visual area for a given hemisphere as described in the preceding text. We then averaged visual responses across voxels within a given visual area, which we call visualresp, and responses from the condition of interest, cresp. Finally, we took the ratio of the two means
 | (1) |
|
RESULTS |
|---|
|
TOPABSTRACTINTRODUCTIONMETHODSRESULTSDISCUSSIONAPPENDIXGRANTSACKNOWLEDGMENTSREFERENCES |
|---|
|
The temporal profile of the response on single-step trials also indicates that the response is driven by the remapped trace of the stimulus rather than by the stimulus itself. In the single-step task, the stimulus appears and stays on the screen for 1,000 ms prior to the cue to make an eye movement. The eye traces indicate that the eyes began to move 200 ms after the tone. The response to the remapped stimulus trace should have a latency that is 1,200 ms longer than the response to the visual stimulus. The example voxel in Fig. 4 indicates that this is in fact the case. The visual response in Fig. 4A begins to rise at 2,000 ms after the onset of the stimulus, consistent with the time course of visually-driven hemodynamic response curves (Boynton et al. 1996). In contrast, the remapped response in Fig. 4B begins to rise at 3,000 ms after the onset of the stimulus. The latency difference between these two curves corresponds to the period between the onset of the visual stimulus and the auditory cue to initiate a saccade.
In summary, the example voxel illustrates four response properties that characterize remapping. This voxel responded to the contralateral visual stimulus in the fixation task, responded to the remapped stimulus trace in the single-step task, did not respond strongly in either of two control conditions, and responded in the single-step task at a latency predicted by the timing of the task. We characterize each of these four response properties below using a Bayesian statistical model of the fMRI time series data. The goal of this analysis is to quantify the degree to which these four properties are present in each visual area across the group of subjects.
Responses to the remapped stimulus trace
The central question in this experiment is whether early- and intermediate-level visual areas respond to the remapped trace of the stimulus. As part of our voxel selection criteria, we identified voxels that responded to direct, contralateral visual stimulation. In this section, we ask whether these same voxels also respond to the remapped trace of a stimulus presented in the ipsilateral visual field (Fig. 1A). Remapped responses were measured on trials of the single-step task in which the stimulus appeared in the ipsilateral visual field and a subsequent eye movement brought the recently stimulated screen location into the contralateral visual field. For each voxel, we estimated the magnitude of the responses in the single-step task, sstepresp, and the posterior probability that the responses were greater than zero given the data, P{sstepresp > 0|Y}, where "resp" denotes response magnitude and "sstep" refers to the single-step task.
We observed robust activity during the single-step task in each visual area. The majority (60%) of the hV4 voxels that exhibited a visual response also exhibited a response in the single-step task that reached a q 0.95 posterior probability threshold (Fig. 5A, dark gray bars). A substantial proportion of visually responsive voxels in V3A (43%) and V3 (35%) exhibited responses in the single-step task. We observed responses in the single-step task in only a about a quarter of visually responsive voxels in V2 (26%) and V1 (22%).
|
This monotonic relationship between response strength and position in the visual hierarchy is demonstrated by a series of pairwise comparisons. These comparisons were computed using Monte Carlo simulations (see METHODS). The largest response was observed in hV4. There was a high posterior probability (q 0.95) that the response in hV4 was larger than responses in V1–V3, and there was a high posterior probability (q = 0.64) that the response in hV4 is larger than the response in V3A. The next largest response was observed in V3A. There was a high posterior probability that this response is larger than responses in V1–V3 (q 0.91). The third largest response was observed in V3. There was high posterior probability (q 0.95) that this response is larger than responses in V1 and V2. Finally, the response in V2 was only marginally larger than the response in V1 (q = 0.63). This series of comparisons indicates that the strength of responses in the single-step task increases at each successive stage in the hierarchy.
These results indicate that the single-step task activates higher-order visual areas more strongly than early visual areas. This difference was evident in both number of activated voxels (prevalence) and the relative response strength (magnitude) across areas. As will be described in the following text, responses in the single-step task reflect activity due to the stimulus and to saccades, as well as to remapping. In subsequent sections, we perform several additional analysis aimed at isolating activity associated with remapping.
Responses in control conditions
As illustrated in the single-voxel example, both ipsilateral stimuli and saccades evoke small responses in visual cortex (Fig. 4, C and D). It is therefore possible that a portion of activity in the single-step task could be attributed to either the ipsilateral stimulus or to saccades alone rather than to remapping activity per se. In this section, we analyze activity in the two control conditions to determine the degree to which ipsilateral stimuli and saccades contributed to activity in the single-step task.
RESPONSES TO IPSILATERAL VISUAL STIMULI ALONE.
In the single-step task, a visual stimulus flashes in the ipsilateral visual field. Although receptive fields in striate and extrastriate cortex are predominantly contralateral, it is conceivable that the ipsilateral stimulus itself elicited a response. This consideration is particularly important in areas hV4 and V3A because receptive fields increase in size at later stages of the visual system and some extend into the ipsilateral visual field (Gattass et al. 1981, 1988; Kastner et al. 2001). In the single-step task, it is possible that neurons with large receptive fields that extended into the ipsilateral visual field could have been driven by the stimulus to a greater degree than V1 neurons that have smaller receptive fields.
We assessed this possibility by measuring responses in a stimulus-only control condition (Fig. 1B). In this condition, subjects maintained fixation while a stimulus flickered in the ipsilateral visual field. This condition was balanced with the single-step task for orbital position and visual stimulation. The only difference between the two conditions was the presence or absence of the auditory cue and the resultant saccade. Fewer than 10% of visually responsive voxels in V1 and V2 responded to the ipsilateral stimulus with a posterior probability that reached a q 0.95 threshold (Fig. 5A, light gray bars). This observation indicates that the ipsilateral visual stimulus did activate a small proportion of voxels in each visual area. Ipsilateral responses were slightly more prevalent in areas V3, V3A, and hV4, with between 10 and 13% of voxels reaching threshold in each area. Although these responses in the stimulus-only condition reveal that some voxels in each area are activated by the ipsilateral stimulus, such responses do not account for the activation we observed in the single-step task. Ipsilateral responses were far less prevalent in the stimulus-only condition than in the single-step task.
We considered whether there were differences across visual areas in the magnitude of ipsilateral responses (Fig. 5B, light gray bars). Areas V3A and hV4 had the largest ipsilateral response (median values of 7 and 5% of the contralateral visual response). The posterior probabilities that either of these responses were larger than responses in any of the other cortical areas were slightly greater than chance (0.50 < q < 0.75). These comparisons indicate a small increase in the magnitude of ipsilateral responses in later stages of the hierarchy. Even the largest responses, however, were small relative to visual responses in these same voxels. We conclude that ipsilateral responses are both too weak and too sparse to account for the relatively large responses observed in the single-step task.
RESPONSES TO SACCADES ALONE.
A potential concern is whether saccades alone activate visual cortex. This issue is particularly important in higher-order visual areas. Neurons in both V4 and V3A fire in relation to saccades directed toward their visual receptive fields (Nakamura and Colby 2000; Tolias et al. 2001). Furthermore, differences between areas in receptive field size could have increased the chances of observing saccade-related activity in V3A and V4 relative to other visual areas. The logic is as follows. In the single-step task, subjects made 16° saccades. Thus visual responses associated with processing the saccade target were located in the 16° representation in the cortical retinotopic map. Because receptive fields are larger in V3A and hV4, the 16° representation is more likely to overlap with the expected site of remapped activation (9°) in V3A and hV4 than in area V1. It was therefore critical that we determine the extent to which saccades contribute to responses in the single-step task.
We addressed this issue by testing subjects on a saccade-only control condition (Fig. 1C). This condition was balanced with the single-step task for orbital position, auditory stimulation, and number of saccades. The only difference between the two conditions was the presence or absence of the visual stimulus in the 1 s preceding the cue to initiate a saccade. We found that saccades in the absence of the visual stimulus did activate voxels in each visual area (Fig. 5B, medium gray bars). A minority of visually responsive voxels in V1 and V2 (12 and 14%) responded in the saccade-only condition with a posterior probability that reached a q 0.95 threshold. Saccade-related responses were more prevalent in areas V3 (20%), V3A (24%), and hV4 (22%) than in V1 and V2. This analysis indicates that saccades may have contributed to activity in the single-step task. Moreover, the contribution of saccade-related activity was larger than the contribution of ipsilateral visual responses. However, responses in the saccade-only condition were still less prevalent than in the single-step task (Fig. 5A, medium vs. dark gray bars), indicating that responses in the single-step task cannot be attributed primarily to saccades.
We considered whether there were differences in the magnitude of saccade-related responses across visual areas (Fig. 5B, medium gray bars). The largest responses in the saccade-only control condition were observed in areas V3A and hV4 (median values of 24 and 16% of the visual response, respectively). The responses in V3A and hV4 were not statistically different (q = 0.53). Neither V3A nor hV4 had a high probability of being larger than responses in V3 (q < 0.80). However, there was a high probability (q 0.95) that responses in V3, V3A, and hV4 were all larger than responses in V1 and V2. Finally, responses in V1 and V2 were not different from each other (q < 0.5). These pairwise comparisons indicate that saccades activate high-order visual areas to a greater extent than V1 and V2.
In conclusion, the analysis of responses in the saccade-only control condition revealed that saccades in the absence of a salient stimulus do elicit responses. Furthermore, the pattern of activity is similar to that observed in the single-step task in that responses are strongest in higher-order visual areas. Across all visual areas, saccade-related responses were smaller than responses in the single-step task, indicating that saccades alone do not account for remapping activity measured in the single-step task.
Responses in single-step task are larger than the sum of control responses
Analysis of responses in the control conditions indicates that both ipsilateral stimuli and saccades activate striate and extrastriate visual areas to some degree. Activation in the single-step task potentially reflects three factors—visual, saccade, and remapping. We asked whether responses in the single-step task could be accounted for by the simple linear summation of responses to ipsilateral visual stimuli and saccades as measured in the two control conditions. There are two possible outcomes to this analysis. If responses in the single-step task equal the summed activity in the two control conditions, we can conclude that the presence of remapping is small or nonexistent. If, on the other hand, responses in the single-step task exceed the summed activity in the two control conditions, we would conclude that remapping is present in early visual cortex, despite the presence of nontrivial activity in the control conditions.
Selectivity indices provide a convenient method for visualizing the relative strength of responses evoked by sets of conditions. We calculated a three-way selectivity index, S, as follows. For each of the three conditions, c, we calculated
 | (2) |
c is the proportional signal change for a given condition averaged over all visually responsive voxels in a given cortical area. Here, the subscript, i, denotes the ipsilateral version of each of the three task conditions. Selectivity values, Sc, sum to 1 
and are all nonnegative.
The results from this three-way selectivity index are represented on triangular simplex plots (Fig. 6). Position in the simplex was determined as follows. Let Vsstep = (0, 
3/2), Vmac = (–,0), Vstim = (,0) be the vertices of a triangle. The plotted positions of a hemisphere in the simplex are given by 
Position in the simplex represents the degree to which the MR response is selective for each of the three conditions. For example, a voxel that responds most strongly on single-step trials relative to the two control conditions will be located in the top sector. A voxel that responds equally strongly in all three task conditions will be represented in the middle of the simplex.
|
The position of each point in the simplex is determined by the mean of the estimated response. There is a degree of uncertainty regarding each position that is reflected by the posterior SD of the estimate. For example, responses from a given hemisphere would be located in the top sector of the simplex plot if the mean of the posterior distribution in the single-step task was large relative to the mean in the two control conditions. However, there would be low certainty regarding the position of that hemisphere in the simplex if the SD of each distribution was also large. We calculated the following probability that takes this uncertainty into account. From the distribution of the response parameters, remap, sac, and stim, under the Bayesian model, we derived the posterior probability that responses fell above the summation line
 | (3) |
0.95) indicate strong evidence for remapping. Probability values near chance (q = 0.5) indicate that remapping and linear summation cannot be distinguished.
A substantial proportion of voxels exhibited responses in the single-step task that were larger than the sum of responses in the two control conditions. The Bayesian analysis revealed that 43% of voxels in area hV4 had a high posterior probability (q 0.95) of exhibiting a stronger responses in the single-step task in than in both control conditions combined. About one-fifth of voxels in V3 (21%) and V3A (20%) and fewer than one-fifth of voxels in V1 (15%) and V2 (14%) reached this same threshold. The analysis of linear summation indicates that remapping is considerably less prevalent in early visual areas than in area hV4. This analysis of linear summation provides strong evidence for the existence of remapping in area hV4. For reasons that will be discussed in the following text, this analysis may be overly conservative.
Remapping and the subadditivity of hemodynamic responses
In many experimental contexts, it is reasonable to assume that the hemodynamic response elicited by two neural events is equal to the sum of responses measured independently—the hemodynamic response function approximates a shift-invariant linear system (Boynton et al. 1996). However, there is a growing consensus that hemodynamic responses behave nonlinearly in specific contexts (Birn and Bandettini 2005; Birn et al. 2001; Friston et al. 1998, 2000; Huettel and McCarthy 2001; Vazquez and Noll 1998). For example, many studies have shown that there is a saturating nonlinearity for closely spaced neural events (review in Wager et al. 2005). Two visual stimuli that occur in rapid succession evoke an MR response that is smaller than would be predicted by the sum of responses to the two stimuli in isolation. In other words, closely spaced events can sum sublinearly. This effect can result in as much as a 50% decrease i.e., It is not yet known whether sublinear summation of responses reflects true nonlinearities in the hemodynamic response function, nonlinearities in the stimulus-induced neural response, or a combination of the two. A saturating nonlinearity could have affected responses in the single-step task. In this task, two events—a stimulus and a saccade—occur in rapid temporal succession. The resultant response may thus be smaller than would be predicted by the sum of responses to the stimulus and saccade measured in isolation. This nonlinearity would cause the analysis of linear summation in the previous section to be overly conservative.
The issue of hemodynamic nonlinearity is difficult to address directly. The neural phenomenon of remapping is itself nonlinear—the conjunction of a stimulus and a saccade produces a response that is not there if either occur alone. Our approach to estimating the sublinear summation present in our data was to analyze responses in three new conditions. Specifically, we analyzed responses in the single-step task when the stimulus appeared in the contralateral visual field and was followed by a contraversive saccade, the stimulus-only fixation task when the stimulus appeared in the contralateral visual field, and the saccade-only condition in which the saccades matched the saccades on single-step trials. These three conditions mirrored the three experimental conditions in that the stimulus was located in the contralateral rather than the ipsilateral visual field, and the saccade was contraversive rather than ipsiversive.
The logic behind this analysis is as follows. By definition, we did not expect remapping in the contralateral hemisphere. If responses sum linearly, contralateral responses in the single-step task should simply reflect the sum of responses in the stimulus-only and saccade-only conditions
 | (4) |
 | (5) |
by subtracting the estimates 
and 
from 
. But if instead the responses combine subadditively, this will underestimate the remapping contribution. To account for this potential subadditivity, we use the contralateral responses as a control to estimate the degree of subadditivity. Although the combination of closely spaced responses will in general be linear, we approximate the subadditivity by a linear shrinkage of the sum. That is, we write sstepc = u(saccc + stimc), where 0 
u 1. (Constraining u to the interval [0,1] precludes the possibility of superadditive summation.) We thus estimate the parameter u by
 | (6) |
after subtracting the corrected combination of and 
 | (7) |
a conservative estimate of the remapping contribution.
In a final analysis stage, we used the Bayesian model to calculate the posterior probability that is larger than 0, conditional on the subadditivity parameter
 | (8) |
We quantified the degree of sublinear summation in the contralateral hemisphere by calculating the parameter û. As described in Eq. 6, small values of û indicate large subadditivity. The majority of voxels in each visual area had û values that were <1, indicating that contralateral responses in the single-step task were smaller than predicted by linear summation of activity in the control conditions (Fig. 7A). We found that û was smallest in area V1 (median û = 0.76), and was largest in hV4 (median û = 0.96). This is notable because hemodynamic nonlinearity has been studied most extensively in primary visual cortex. Our results suggest that the degree of nonlinearity is variable across voxels and across with cortical areas.
|
, in the ipsilateral hemisphere, according to Eq. 8. This estimate reflects the relative magnitudes of responses in all six trial types (3 conditions, 2 directions). A substantial proportion of voxels in each visual area had nonzero values (Fig. 7B). This estimate of remapping strength is less conservative than the simple linear analysis because the scaling factor, û, reduces the size of the summed visual and saccade activity. Finally, we used the Bayesian model to calculate the probability that was larger than zero, conditional on û (Fig. 7C). The analysis of sublinear summation provides three important insights. First, it indicates that responses did sum sublinearly and that nonlinearities are important to consider when measuring responses to rapidly occurring events. Second, this analysis indicates that remapping is present throughout occipital cortex. Third, this analysis reveals a monotonic relationship between the magnitude of remapping and position in the visual hierarchy.
Time course of visual and remapped responses
Remapping occurs at various points in time relative to saccade initiation. A substantial proportion of neurons in parietal and extrastriate cortex remap predictively, while others begin to respond around the time of the saccade (Duhamel et al. 1992a; Nakamura and Colby 2002). Predictive responses occur at a latency that is shorter than the typical visual response for that neuron. A subset of cells with predictive responses begin to respond to the stimulus trace even before the eyes have moved.
This neural variability in the timing of remapping relative to the saccade should not be observable in our fMRI data. The hemodynamic response function acts as a low-pass filter of the underlying neural activity, obscuring small temporal variations in response onset time. We did expect to observe a difference in response time between visual and remapped activity. In the fMRI version of the single-step task, the stimulus appears and stays on the screen for 1 s prior to the auditory cue to make an eye movement. We expect that remapping occurs around the time of the eye movement. Subjects in this study had a mean saccadic reaction time of 255 ± 128 ms. Thus the onset of the stimulus preceded the onset of the saccade by an average of 1,255 ms, relative to the start of the trial. Because of this interval, remapped responses driven by the stimulus trace should begin 1,255 ms later than visual responses driven by the stimulus. This statement assumes that remapping occurs around the time of the eye movement.
We used the Bayesian estimates of response profile to test this prediction. The three relevant model parameters are lag, attack, and decay. The first two parameters—lag and attack—correspond to early stages of the response. Lag corresponds to the time from stimulus onset to the start of the hemodynamic response; lag is equivalent to response latency. Attack corresponds to the time from the start of the response to the peak of the response; attack is a measure of the rate at which the response rises. Finally, the decay parameter corresponds to the duration from the peak of the response to the point that the response returns to baseline levels; decay measures the duration of response offset.
We predicted that remapped responses would have a longer lag and attack than visual responses but that the two response types would not differ in decay. Our analysis of response profile was performed on the subset of voxels in each visual area in which both visual and remapped responses had a high probability of being nonzero (q 0.95). This selection criterion was necessary because inferences regarding response profile can only be based on actual responses. The analysis described in this section includes a smaller proportion of visually responsive voxels that the analyses described in previous sections because, as we showed in Fig. 5, only a subset of visually responsive voxels in each visual area responded to the stimulus trace.
ANALYSIS OF TIME-TO-PEAK (LAG+ATTACK).
We calculated response peak by simply summing the lag and attack parameters (lag + attack). Combining the lag and attack parameters gives a measure of the time-to-peak relative to the onset of the stimulus. We compared estimates of time-to-peak for visual responses and remapped responses. Our goal was to determine if the difference in time-to
, onset time of the visual (- - -) and remapped response (—). C: stimulus-only control. Presentation of the stimulus in the ipsilateral visual field does not elicit a response in the absence of a saccade. D: saccade-only control. The saccade alone does not elicit a strong response in the absence of a stimulus. Each curve was estimated from responses on 64 trials. 
, 1 SE.
