INTRODUCTION

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

Whenever we shift our gaze, the image of an object that remains fixed in the world moves across our retinae. Under these circumstances, we do not perceive object motion but instead correctly sense that the object remains still. The ability of the visual system to account for self motion while gauging motion in the world is called space constancy (Bridgeman 1995; Deubel et al. 1998; Helmholtz 1962; Holst and Mittelstaedt 1950; Stark and Bridgeman 1983). The ability to correct internal representations for one's own movements is also critical for goal-directed behavior, especially if the goal disappears and gaze is shifted prior to goal acquisition. Although goals do not normally disappear in conjunction with a gaze shift, shifting the fovea away from an object causes its image to move into a portion of the visual field where resolution is relatively poor. Under these circumstances, the sensorimotor transformation used to acquire the world-fixed target may rely on spatial memory rather than on visual information (Karn and Hayhoe 2000). If a gaze shift occurs after a target has been memorized, then the gaze shift will have changed the eye-centered position of the goal, and therefore the saccade plan to acquire that target must be adjusted. Humans and primates make these adjustments well, deriving accurate spatial estimates from a combination of visuospatial memory with vestibular and proprioceptive postural signals (Bloomberg et al. 1991; Israel et al. 1993; Medendorp et al. 1999, 2002; Mergner et al. 1998; Philbeck et al. 2001) and copies of efferent oculomotor commands for saccadic (Hallett and Lightstone 1976; Karn et al. 1997; Sparks and Mays 1983) and pursuit eye movements (Herter and Guitton 1998; Honda 1990; Ohtsuka 1994; Schlag et al. 1990; Zivotofsky et al. 1996).

Because many objects are fixed in the world, it is possible that the visual system generalizes and treats all objects as world-fixed. Several observations are consistent with this idea and support an emerging view that internal spatial representations are adjusted whenever a gaze shift occurs. Lesions in the posterior parietal cortex (PPC) impair the ability of both monkeys (Li and Andersen 2001) and humans (Duhamel et al. 1992b; Heide et al. 1995; but see Philbeck et al. 2001) to make compensatory adjustments for gaze shifts. Electrophysiological studies in the monkey have revealed several brain regions in which single neurons adjust their activity during memory-period gaze shifts in a manner consistent with storage of world-fixed positions, including the lateral intraparietal area (LIP) (Duhamel et al. 1992a, Goldberg and Bruce 1990; Mazzoni et al. 1996, Powell et al. 1997), frontal eye field (FEF) (Goldberg and Seagraves 1989, Umeno and Goldberg 1997), V3A (Nakamura and Colby 2000, 2002), the parietal reach region (Batista and Andersen 2001), and the superior colliculus (Sparks and Mays 1983, Walker et al. 1995). A trans-thalamic pathway has been identified by which copies of efferent saccadic signals from the superior colliculus (SC) are conveyed to FEF for the purposes of adjusting spatial representations for gaze changes (Sommer and Wurtz 2002). While these studies show where in the brain adjustments for self motion are likely to occur, they have not provided any quantitative treatment of how well these adjustments are performed. Furthermore, none of these studies address whether or not the brain bypasses these adjustments under certain conditions, such as when an object is perceived as moving with gaze.

There are numerous natural circumstances under which referring spatial information to an eye-centered frame may be more appropriate for perception and action than a world-centered reference frame. During smooth pursuit of a large rigid object, for example, every feature on the object is fixed with respect to gaze, even those features that are not being foveated. To inspect one of these peripheral features, a saccade must be directed toward a target that is moving in the world but fixed on the retina. An accurate saccade requires that the visual system ignore the change in eye position that occurs as a result of the smooth-pursuit tracking because both the stimulus for that sensory-motor transformation (the retinal signal) and the output of the transformation (the saccadic command) are eye-centered. Any adjustment for the pursuit-driven gaze change would result in an inaccurate saccade. Generally, when remembering the position of a gaze-fixed object, the visual system should ignore gaze shifts that occur once the object's position has been memorized.

We set out to test whether the visual system treats memorized positions differently depending on the reference frame in which they are fixed. Animals were trained to memorize the location of a transient visual stimulus, to track that location as either gaze- or world-fixed during a memory-period gaze shift, and then to provide a readout of that memory by making a saccade to the remembered location. If the visual system stores spatial memories in world-centered coordinates or treats objects as world-fixed by default, then remembering the location of a gaze-fixed object would require an additional operation each time a gaze shift occurs, relative to remembering a world-fixed location. This additional operation would be expected to add noise to the stored signal (Karn et al. 1997). As a result, spatial memories for locations that move with gaze would be degraded, and reports of remembered gaze-fixed locations would be less precise and perhaps less accurate than reports of remembered world-fixed locations. In contrast, if the brain treats object locations flexibly based on experience, then remembering a gaze-fixed location would require fewer operations than remembering a world-fixed location, and reports of remembered gaze-fixed locations would be more precise and perhaps more accurate than reports of remembered world-fixed locations.

Of these two models, our results are most consistent with a flexible, eye-centered storage mechanism. After either a whole-body rotation or smooth-pursuit eye movement, saccades to memorized targets were equally accurate but more precise for gaze-fixed than for world-fixed locations. However, after a visually guided saccade, neither precision nor accuracy were affected by the reference frame of the memorized stimulus. These findings suggest that the visual system may engage different mechanisms to update stored signals depending on how gaze is shifted.

	METHODS

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

Experiments were designed to compare the properties of saccades directed to memorized target locations under a variety of conditions. Two adult monkeys (Macaca mulatta, M1 and M2) were trained to memorize target locations and to imagine that these memorized locations were either fixed in the world (world-fixed) or moved along with their gaze (gaze-fixed). After memorizing a target location, gaze relative to the world was shifted by a passive whole-body rotation, an active smooth-pursuit eye movement, or an active saccadic eye movement. Shortly after the gaze shift ended, animals were cued to saccade to the remembered target location. Saccade accuracy, variability, latency, and velocity were measured and compared under the various conditions. Single-unit recordings from the lateral intraparietal area accompanied all psychophysical measurements reported here and will be described in a separate report. All procedures conformed to the Guide for the Care and Use of Laboratory Animals (ISBN 0-309-05377-3) and to National Institutes of Health and Washington University IACUC guidelines.

Surgery

Monkeys were fitted with a titanium prosthetic device to stabilize the head and a single scleral search coil for chronic eye-movement recording (Robinson 1963). The stabilization device was custom machined from titanium and fastened to the skull using titanium screws. The eye coil, consisting of three loops of Teflon-insulated flexible multi-strand wire (Cooner Wire, Chatsworth, CA), was surgically implanted under the conjunctiva (Judge et al. 1980). Sterile surgery was performed under inhalation anesthesia (isoflurane, 0.5-2.0%). Postoperative analgesics were provided as necessary.

Experimental apparatus

During experiments, the monkey was seated in a Lexan box (Crist Instruments) that was mounted on a vestibular turntable (Neurokinetics). The axis of the turntable was earth-vertical and was centered between the animal's ears. Eye movements were monitored using earth-mounted 4-foot rectangular field coils (CNC Engineering). Visual stimuli were projected (Electrohome, Model ECP 4100) onto a 100 × 80-cm screen placed 58 cm from the animal. The room was otherwise completely dark, and this was confirmed by a dark-adapted human observer. All aspects of the experiment, including visual stimuli, turntable rotation, eye-position recording, and behavioral control, were computer controlled (custom software). Animals were monitored at all times using an infrared camera and illuminator. Eye and turntable position was logged every 2 ms along with information regarding the trial type, target location, window sizes, and the timing of each event. Visual stimulus presentation times were known to within one video refresh (17 ms).

Experimental design

INITIAL FIXATION AND TASK INSTRUCTIONS. Figure 1 illustrates the timing and experimental paradigm. At the start of each trial, the animal had 1,500 ms to acquire the fixation spot (0.26 × 0.28°, fixation tolerance: within 2-3°). The color of the fixation spot provided the first of two instructions informing the animal whether an upcoming target would be gaze-fixed (purple) or world-fixed (yellow). One hundred milliseconds after initial fixation, the second of these task instructions occurred in the following way. Two vertically oriented flanking bars (0.4 × 10°) appeared 7° to either side of fixation. After 250 ms, the fixation spot slid to the right or left (10°/s for 300-500 ms) and then disappeared. (In rotation trials, the chair also rotated along with the fixation spot.) If the flanking bars moved with the fixation spot, then the target in the upcoming trial would be gaze-fixed. If the flanking bars remained fixed in the world, then the target in the upcoming trial would also be world-fixed.

View larger version (42K): [in this window] [in a new window]   Fig. 1. The memory saccade paradigm. A: time course. A fixation point appeared, followed by an instructional sequence (see C). After an additional 500 ms, a peripheral target was flashed for 150 ms. Next came a memory period, during which gaze was perturbed by a whole-body rotation, smooth pursuit or saccadic gaze shift (see B). Finally, the fixation point was extinguished, and the animal was rewarded for performing a memory saccade to the remembered target location. The target reappeared 300 ms after the target location was acquired. Up to 2 rewards were then given, depending on how close the animal was to the target. B: velocity and position trajectories of stimuli and gaze during the 3 types of gaze shifts. C: spatial layout of the trials. An instructional sequence conveyed whether the stimulus would be world- or gaze-fixed (far left). Two flanking bars appeared on either side of the fixation point, and then the fixation point moved slowly for 5°. On world-fixed trials (top), the flanking bars remained fixed in the world. On gaze-fixed trials (bottom), the flanking bars moved with the fixation point. The stimulus (near left) could appear at 1 of many different locations. It disappeared, and gaze was shifted to either the left or right (middle). After the memory period (near right), the fixation point disappeared and the animal was rewarded for moving to the correct spatial location (far right).

PERIPHERAL TARGETS AND GAZE SHIFTS. An additional 500 ms followed the instructional sequence before a purple flash (2 × 2°) appeared in the periphery for 150 ms and then disappeared. Peripheral targets could appear with a horizontal displacement relative to the fixation spot of 0, ±10, ±20, or, in some sessions, ±30 and ±40°, and a vertical displacement of 4, ±10, or ±18°. There were a total of 25 and 45 possible target locations for M1 and M2, respectively.

FINAL MEMORY PERIOD AND MOVEMENT TO THE REMEMBERED LOCATION. After the gaze shift, all trials had a final memory period before the animal was cued to move. In M1, this period was 750-1,250 ms for saccade trials and fixed at 400 ms for pursuit and rotation trials. In M2, this period was 600-1,200 ms for all perturbation types. The fixation spot was extinguished at the end of this memory period, cuing the movement to the remembered target location. Trials in which fixation was broken prior to the offset of the fixation spot were discarded and not analyzed further. Feedback to the animal after such an error was provided by a bright flash and a 1-3 s time-out.

DATA ANALYSIS. We used two alternative measures of behavioral performance. The first measure was the end point of the first saccade after fixation point offset. The second measure was the eye position in the last 50 ms prior to target reillumination. Only the latter measure included corrective saccades (performed in the absence of visual feedback). Both absolute eye-in-head position as well as change in eye position were analyzed with only the latter taking account of the small deviations in eye position from the fixation target immediately prior to the saccade. To eliminate outliers that did not represent reasonable attempts to acquire the remembered target position, we first excluded trials in which animals missed the ideal target location by >20° vertical or >30° horizontal (0.8% of trials) or in which saccades were abnormally slow (2.6% of trials). It is possible that large horizontal errors could have resulted from an animal mistaking which reference frame to use (gaze- or world-fixed). To eliminate these outlying errors, most analyses excluded trials where the memory-guided saccade fell >12.5° from the ideal target location (an additional 5.2% of trials; Figs. 3-6, Table 2). Finally, data are presented using only 20° gaze perturbations and saccades to targets within the central 20° of gaze. Thirty-degree target trials were performed more often under world- than gaze-fixed conditions and were therefore excluded from the primary analysis. For each analysis, the results were similar whether or not these exclusion criteria were applied unless stated otherwise.

Additional design considerations

BEHAVIORAL STRATEGY. We hoped that the large number of trial types (2 reference frames, 1 or 2 shift amplitudes, 2 shift directions, 3 shift types, 25 or 45 target positions) would force the animals to compute final target location on each trial. An alternative strategy would be to recognize each conjunction of stimulus conditions as a unique case and to associate a learned response with each particular conjunction. While a full treatment of the subtle distinctions between computational and associative strategies are beyond the scope of this report, we did perform several control experiments to address the issue. One animal was presented with several targets at novel locations, and, in a second series of experiments, whole-body rotations of novel amplitudes or speeds. The animal responded appropriately on many (though not all) of these trials, suggesting that some degree of computation was in fact occurring.

BLOCKING. For both animals, at least two types of gaze shifts were interleaved at all times (e.g., pursuit and whole-body rotation). In M1, world- and gaze-fixed trials were segregated into blocks of 32-64 trials. At the start of each block, 5-12 instruction trials were delivered. Here the peripheral target was not extinguished but instead dimmed shortly after its appearance to near-threshold brightness. On these instruction trials, the animal could see that the target was either remaining fixed in the world (signaling the start of a world-fixed block) or else moving along with the animal's gaze (signaling the start of a gaze-fixed block). Of course, the animal could ignore the target, locating it only after the fixation point was extinguished. We reasoned that because the dim target was difficult to detect, the animal would covertly track the target during instruction trials and thereby learn to use either the world or gaze as the frame of reference. Saccade performance during instruction trials are of course excluded from this report.

	RESULTS

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

Experiments were designed to test whether saccades made to remembered world-fixed target locations were similar to saccades made to remembered gaze-fixed target locations. We first asked whether animals would be able to remember gaze-fixed locations at all, and whether they could switch from one reference frame to the other at will. To determine this, animals memorized the location of a visual stimulus. After the stimulus disappeared, their gaze was shifted by a visually guided saccadic eye movement, a smooth-pursuit eye movement, or a whole-body rotation accompanied by suppression of the VOR. In this report, we present only data collected while single neurons were recorded to facilitate behavioral-neuronal comparisons. Single-neuron data, and therefore the data reported here, were collected using only 20° gaze shifts because these larger shifts maximized the difference in final target location between world- and gaze-fixed conditions. These data were collected over 32 (M1) and 35 (M2) sessions over a period of 1 yr in each animal.

Animals can remember both world- and gaze-fixed locations

Table 1 shows that trained animals were able to remember target locations that either remained at a fixed position in the world or moved along with gaze. Table entries show the percentage of trials completed successfully (see METHODS for criteria) and the number of trials on which this percentage is based, broken down by animal and type of gaze shift. Performance was good (75% correct or better) under most conditions with no clear preference for one reference frame over the other. In particular, both animals remembered world-fixed targets better after saccadic gaze shifts (see following text for details) but remembered gaze-fixed targets better after rotational gaze shifts. Results on pursuit trials were mixed with better performance on gaze-fixed targets in one animal and on world-fixed targets in the other animal. The fact that success rates were >70% for all but gaze-fixed saccadic shift trials in one animal suggests that the animals were able to remember target locations that were either world- or gaze-fixed.

If either world- or gaze-fixed targets are in some sense natural or hard-wired, then we would expect a clear preference to appear during early training. This was not the case. Both animals were first trained on rotation trials. Initially, world- and gaze-fixed trials were performed on separate days (prior to data collection). However, there was no clear performance advantage for one reference frame in either animal. Later, world- and gaze-fixed trials were performed in separate blocks on the same day, and finally, for M2, trial types were fully interleaved. Animals were next trained on pursuit trials. In neither animal did any consistent pattern emerge to suggest that they preferred one reference frame over the other.

After several months of neuronal recording during the rotational and pursuit tasks, world- and gaze-fixed double saccade trials were introduced. M1 immediately learned to perform short blocks of world-and gaze-fixed double saccade trials. (Each block contained rotation plus pursuit or rotation plus saccade trials, with all targets either world- or gaze-fixed. See Additional design considerations). We attempted to train M2 on a fully interleaved version of the task, but performance was poor. At this time, M2 had difficulty making large leftward saccades and so often fell short of both world- and gaze-fixed targets. This difficulty occurred even on practice trials in which the target remained visible and the trial required only a saccade to a visible target, strongly suggesting that the difficulty lay in the animal's motor performance and not in the computational aspects of the task. Errors on leftward targets seemed to confuse the animal and often led to a string of errors on subsequent trials. These strings of errors account for the bulk of the errors reported in Table 1 [M2, saccade (interleaved)]. When the animal was retrained to use only a single reference frame for saccadic gaze shifts, performance went up sharply and was similar for the two frames of reference [Table 1, saccade (blocked)]. It is worth noting that this equalization of performance is not simply due to the animal no longer having to switch between reference frames. The blocks containing targets in a single reference frame for saccadic trials also contained rotation trials in which both world- and gaze-fixed reference frames were interleaved. The data from successful blocked and interleaved trials were similar, and therefore these trials are combined for the purposes of this report.

Distribution of saccade end points to world- and gaze-fixed targets

Although success rates provide a rough estimate of task performance, a more appropriate way to gauge the animal's spatial memory is to examine the distributions of saccade end points under world- and gaze-fixed conditions. Figure 2 shows the end points for saccades to gaze- and world-fixed targets from M2 after whole-body rotation and saccadic gaze shifts. Each histogram represents the frequency distribution of horizontal saccade amplitudes for one of three ideal horizontal saccade amplitudes (0°, leftward 10°, rightward 10°) and one of two gaze shifts (leftward 20° or rightward 20°). The data show that animals performed well with saccade end points distributed close to the veridical target position under each condition. For targets with different vertical amplitudes, and when gaze was shifted by different amounts (10 or 20°), saccades were clearly directed toward the appropriate locations (data not shown). All these observations suggest that animals were taking into account both initial stimulus location as well as the direction and amplitude of the gaze shift to calculate the final position of the remembered target.

View larger version (31K): [in this window] [in a new window]   Fig. 2. Saccades to remembered locations after gaze shifts in M2. Frequency distributions of horizontal saccade amplitudes for trials with 3 different ideal saccade end-point locations. Solid (dashed) lines indicate the horizontal component of memory saccades obtained after a leftward (rightward) gaze shift. Colors indicate whether the ideal saccade end point was located above and 10° to the left (red), directly above (green), or above and 10° to the right (blue) of the fixation point. Initial target locations relative to the fixation point are indicated by triangles below the abscissa; the 30° initial target locations used in world-fixed conditions are not marked (bottom). Final ideal target locations, relative to the fixation point, are indicated by thin vertical lines above the abscissa. For gaze-fixed trials (top), initial and final target locations are aligned. For world-fixed trials (bottom), initial and final target locations are offset from one another by the amplitude of the gaze perturbation (20° to the right or left). Actual saccade end points fall close to the ideal end points after both rotational (left) and saccadic gaze shifts (right). The distributions are narrow for gaze-fixed targets after a rotational shift (top left); intermediate for both gaze- and world-fixed targets after a saccadic shift (right, top and bottom), and broad for world-fixed targets after a rotational shift (bottom left).

Despite this generally high level of performance, there was a clear difference between world- and gaze-fixed conditions. After whole-body rotations, animals were not only more often correct on gaze-fixed trials (Table 1), they also showed less variability in their spatial memories for gaze-fixed targets (Fig. 2). Relative to world-fixed distributions, gaze-fixed distributions were narrower and consistently unimodal after whole-body rotations, indicating that the animals were more precise when remembering spatial locations that remained fixed with respect to gaze. One possible explanation of this effect is derived from the observation that the VOR typically undercompensates for gaze shifts (Jell et al. 1988). If underestimation of rotation is a general property of the vestibular system, then one might predict errors in spatial memory on world-fixed rotation trials due to an underestimate of how far gaze has been shifted. However, the data were not consistent with this prediction. Saccade end points after leftward (solid lines) and rightward (dashed lines) shifts were not markedly different from one another. We next quantified the effects of target reference frame on spatial memory by comparing end-point variability after whole-body rotation, smooth pursuit, and saccadic gaze shifts.

Reference frame influences saccade variability after memory-period rotation or pursuit but not after memory-period saccades

To quantify saccade end-point variability, we calculated the SD of the horizontal change in eye position for different target locations and under different conditions. After whole-body rotations, saccades were more variable for world- than gaze-fixed targets at every location, averaging 50% greater variability on world-fixed trials in M2 (Fig. 3B). For each type of gaze shift, differences between world- and gaze-fixed variability were largely independent of both target location and gaze shift direction. Saccades directed toward the most eccentric targets for a given condition (±20° under world-fixed conditions; ±30° under gaze-fixed conditions) were less variable than saccades to more central targets in the array. This reduction in variability for the most eccentric targets is likely to be an edge effect caused by spatial limitations in the stimulus array. We therefore compared saccade variability for world- and gaze-fixed targets for each type of gaze shift based on data averaged across the three central target locations and the two gaze shift directions. Table 2 presents mean saccade variability for each animal under each experimental condition. In both M1 and M2, the effect of reference frame on saccade variability was highly significant after whole-body rotations and smooth-pursuit gaze shifts. In M1, an effect of reference frame on memory-guided saccade variability was observed after saccadic gaze shifts but with world-fixed conditions showing less variability than gaze-fixed conditions.

View larger version (13K): [in this window] [in a new window]   Fig. 3. Accuracy and precision of saccades to world- and gaze-fixed targets after whole body rotations in M2. A: on average, saccades to targets within 10° of the fovea were quite accurate, whereas saccades to targets 20° from the fovea were somewhat hypometric. Hypometria was slightly greater for gaze-fixed targets (gray) than for world-fixed targets (black). B: saccade variability was greater under world-fixed compared with gaze-fixed conditions. Variability was measured as the SD of the distribution of horizontal saccade amplitudes and corresponds to the width of the distributions shown in Fig. 2. The difference in variability was largely independent of target location.

Increased variability during world-fixed conditions might occur if the amplitude of saccades to world-fixed targets was much greater than the amplitude of saccades to gaze-fixed targets. However, this was not the case. Saccades to gaze-fixed targets were slightly more hypometric than saccades to world-fixed targets. This is shown graphically for M2's saccades after whole-body rotations (Fig. 3A). Across all conditions, saccades to 10 or 20° eccentric targets were 17% hypometric in M1 and 12% hypometric in M2.

Another possible explanation for increased variability during world-fixed conditions is that systematic underestimates in the size of the gaze shifts led to biased estimates of individual world-fixed locations, and when combined across gaze shift directions, these biased estimates resulted in increased variability. To test this possibility, we recalculated the SDs of saccade end points after rotation and pursuit gaze shifts without combining across gaze shift directions. There were eight separate comparisons (2 animals × 2 directions × 2 types of gaze shift). In every case, saccades to world-fixed targets were more variable than saccades to gaze-fixed targets (P < 0.001 for 7 comparisons; P < 0.05 for the 8th).

We also tested whether saccade end-point variability was confounded by other differences in behavior when tracking locations in the two reference frames. For rotation and pursuit conditions, variability was greater for world- compared with gaze-fixed targets regardless of whether the initial saccadic end points or the final eye positions were used and regardless of whether or not error trials were included in the calculations. In contrast, the relative variability of end points to world- and gaze-fixed targets after saccadic gaze perturbations was inconsistent. In most cases, there was no significant difference between world- and gaze-fixed trials; but depending on whether the effects of corrective saccades and the results obtained on error trials were included in the calculations, we found small but inconsistent biases (0.01 < P < 0.05) favoring one reference frame over the other (data not shown). In particular, the reference frame effects on saccadic shift (double-step) trials were weak and inconsistent across the two animals, suggesting that these differences represent either false rejections of the null hypothesis (type II errors) or real effects that were idiosyncratic to an individual animal or to a particular set of circumstances.

To determine which effects were most reliable, we combined variability data from the two animals and repeated our statistical analyses. Figure 4 shows the mean horizontal SD for saccades to world- and gaze-fixed targets after each type of gaze shift. Based on the comparisons that reached statistical significance in the combined analysis (P < 0.05 after correction for 15 multiple comparisons), saccade variability data can be clustered into three tiers. Saccades to world-fixed targets after rotation or pursuit gaze shifts were the most variable; saccades to gaze-fixed targets after rotation were the least variable; and saccades to gaze-fixed targets after pursuit and both world- and gaze-fixed targets after saccadic shifts had intermediate variability. This hierarchy suggests that while gaze-fixed conditions yield more precise estimates under certain conditions, the reference frame alone cannot account for the differences in variability we observed across our task conditions. Rather, how gaze is shifted appears to play as critical a role in determining the precision of spatial memories as the reference frame in which they are maintained.

View larger version (20K): [in this window] [in a new window]   Fig. 4. Precision of saccades to remembered targets. Saccade variability (SD of saccade amplitudes, averaged across the 3 central target locations) is plotted for saccades to world-fixed (black) and gaze-fixed (gray) targets, after a rotational, pursuit, or saccadic gaze perturbation during the memory period. Error bars denote the SE of the values calculated across sessions and locations. Background shading indicates 3 statistical tiers, within which all paired comparisons are NS and between which all paired comparisons are significant (P < 0.05 after correction for multiple comparisons). Data are shown for the horizontal component of memory saccades, averaged across both animals, excluding trials with >20° vertical error or >12.5° horizontal error. This conservative horizontal criterion excludes trials on which the animal may have been using the inappropriate reference frame.

One possible explanation for the reduction in variability for gaze-fixed targets after rotations, as compared with pursuit and saccadic shifts, is that the changes in orbital eye position impact spatial memory processing even when the retinotopic position of the memory is fixed (see DISCUSSION of Fig. 7 for complete treatment). If retinotopic target position and eye position are handled independently, then we might expect variability for world-fixed targets to be increased for conditions when the eyes move in the orbits. Although we did not detect this difference across the full dataset, we conducted a post hoc analysis of variability from trials in which rotation data and smooth-pursuit data were collected in the same behavioral session. Across this restricted dataset (35 sessions), a paired t-test of the horizontal SD obtained from world-fixed trials after rotations and pursuits in the same session revealed a marginally significant effect (P = 0.04), with memory-guided saccades after pursuit eye movements being more variable (3.67 ± 0.13°) than those performed after a whole-body rotation (3.37 ± 0.11°). Although this difference should be viewed cautiously, it is nevertheless consistent with a memory system in which eye position and retinotopic target position impact spatial processing independently.

Saccades were not consistently biased toward either frame of reference

We next examined the accuracy of saccadic responses to gaze- and world-fixed targets. As illustrated in Figs. 1 and 3A, saccades to eccentric targets were typically hypometric. We generated an index of accuracy that accounted for this hypometria, and thus allowed us to ask whether memory-guided responses were biased toward the gaze-fixed or world-fixed position of the stimulus. Table 2 shows the individual subject index values.

After slow gaze shifts, subjects tended to move a few degrees toward the inappropriate reference frame, but neither animal showed a clear bias toward gaze- or world-fixed targets: individual animals showed small differences between gaze- and world-fixed index values, but these effects were small and in opposite directions in the two animals. Similar inconsistencies were seen when corrective saccades or error trials were included in the measurements. Therefore we again combined data across the two animals and compared the effect of reference frame on the accuracy of memory-guided responses (Fig. 5). After correcting for eccentricity-dependent effects (e.g., hypometria), memory-guided responses were highly accurate after a visually guided saccade (within 1°), with no significant difference between world- and gaze-fixed targets. In contrast, memory-guided responses after a rotation or pursuit were less accurate overall (2-3°), with small and inconsistent biases evident in the comparison of gaze- and world-fixed index values. We take from these findings that the visual system's estimate of an object's position is more accurate after rapid (saccadic) gaze shifts than after slow gaze shifts, whether the remembered position is gaze- or world-fixed.

View larger version (16K): [in this window] [in a new window]   Fig. 5. Accuracy of saccades to remembered targets. A: illustration of the accuracy index as calculated for saccades to the +10° target location (see scale at bottom). When the gaze-fixed target location is correct (black circles, top), world-fixed locations are by definition incorrect (gray circles, top), and vice versa for world-fixed conditions (bottom). The mean of the actual saccade end points for a given condition (vertical hashmarks) is separated from the correct location for that condition by a vector (a-d). Errors in the direction of the incorrect target are arbitrarily considered as positive. Because the relative position of correct and incorrect targets depends on the direction that gaze has shifted, mean errors after leftward (a and d) and rightward (b and c) shifts were calculated separately. Values for each of the 3 central end points (10, 0, +10°) were averaged to obtain the accuracy index for that condition. Hypometria and other idiosyncratic biases in the subject's memory-guided responses are cancelled out by this method. A value of 0° under either condition indicates no reference frame related error; a value of 10° would indicate that saccades are directed, on average, to a location midway between the gaze- and world-fixed target locations, whereas a value of 20° would indicate that saccades always land on the incorrect target location. B: accuracy index values, from data combined across the 2 subjects, are shown for saccades to gaze-fixed (gray boxes) and world-fixed targets (black boxes) after rotation, pursuit, and saccadic gaze shifts. Errors tended to occur in the direction of the incorrect target location under both gaze- and world-fixed conditions, although memory-guided saccades were not consistently biased toward either reference frame. Memory-guided saccades were more accurate after saccadic gaze shifts than after slow gaze shifts. *, P < 0.05.

Saccade latency and main sequence characteristics were independent of reference frame

Saccade latency (Fig. 6A) and main sequence characteristics (Fig. 6B) were independent of how gaze was shifted and whether targets were gaze- or world-fixed. Saccade latencies to world- and gaze-fixed targets differed by no more than 10 ms in individual comparisons, and these differences were inconsistent across comparisons and animals. The "main sequence" relationship between peak velocity and saccade amplitude indicates that saccades are highly stereotyped (Bahill et al. 1975). We quantified this relationship for saccades to world- and gaze-fixed targets and, as expected, found the slope to be independent of reference frame (Fig. 6B). Both peak velocity and saccade duration were also found to be independent of reference frame (not shown).

View larger version (14K): [in this window] [in a new window]   Fig. 6. Saccade latencies and main sequence characteristics. A: saccade latencies were generally just <250 ms, and similar for world- and gaze-fixed target conditions in both animals. Latencies were shorter for rotation and pursuit conditions in animal M1. In these cases, the delay after the gaze shift was fixed, so that the animal could anticipate when to initiate the memory-guided saccade and thereby reduce saccade latency. B: the ratio of peak saccade velocity to saccade amplitude (i.e., the slope of a straight line fit to the saccade main sequence) was similar under world- and gaze-fixed target conditions in both animals.

	DISCUSSION

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

In the present study we examined the effects of self motion on spatial location memory. We employed a variant of a behavioral paradigm used in the classical double saccade experiments of Hallet and Lightstone (1976) and subsequently employed by many other investigators to test the visual system's ability to correct for self motion while maintaining a spatial memory (e.g., Bloomberg et al. 1991; Herter and Guitton 1998; Honda 1990; Israel et al. 1993; Karn et al. 1997; Medendorp et al. 1999, 2002; Mergner et al. 1998; Ohtsuka 1994; Philbeck et al. 2001; Schlag et al. 1990; Sparks and Mays 1983; Zivotofsky et al. 1996). These studies have shown that localization is spatially accurate after most gaze perturbations, unless the target to be remembered is presented immediately before or during the gaze shift (which was not the case in the current study) (see Schlag and Schlag-Rey 2002 for review). In our task, subjects memorized the location of a peripheral flash. After a delay, the fixation spot moved to an unpredictable location. Finally, after a second delay, the subjects reported the remembered location of the peripheral flash.

Our design extends previous work in two important ways. First, in most previous studies, subjects memorized each location as if it was fixed in the world. When a gaze shift occurred, a world-fixed location moved relative to gaze, and therefore the eye movement that would bring the memorized target location onto the fovea was different before and after the gaze shift. In this study, we added a novel condition in which subjects memorize the location as if it moved with gaze (Israel et al. 1999). In this gaze-fixed condition, the eye movement that would foveate the remembered target location is not affected by the gaze shift. The gaze-fixed condition serves as a control for the world-fixed condition because only the latter requires that subjects use information about the gaze shift to perform the task. By comparing performance under gaze- versus world-fixed conditions, we can determine the cost, if any, associated with maintaining a spatial memory as fixed in the world. Our second extension was to directly compare the effects of different extraretinal gaze signals on spatial memories within the same subjects. Most studies of spatial updating have focused on the visual system's ability to account for a single type of gaze shift. In the present study, we directly compared spatial memories after whole-body rotations, smooth-pursuit eye movements, and saccadic eye movements in the same subjects under both gaze- and world-fixed conditions.

Our findings can be separated into effects on memory precision (i.e., reliability) and effects on memory accuracy. We observed several effects on the precision of memory: on trials with slow gaze shifts (whole-body rotations or smooth pursuit), memory-guided saccades to gaze-fixed locations were more precise (less variable) than saccades to world-fixed locations; on trials with rapid gaze shifts (visually guided saccades), memory-guided saccades were equally precise under gaze- and world-fixed conditions; and within gaze-fixed conditions, memory-guided saccades were more precise after whole-body rotations than after either visually guided saccades or smooth pursuit. A paired comparison of variability after rotations and pursuits revealed a slight improvement in precision on world-fixed trials when the eyes remained fixed in the orbits. Memory accuracy showed a different pattern. Memory-guided saccades were most accurate after visually guided saccades and less accurate after rotations and pursuit eye movements, but accuracy was not influenced by whether the location was memorized as world- or gaze-fixed. Finally, neither the latency nor the dynamics of memory-guided saccades varied as a function of either reference frame or shift condition.

These observations suggest that retinal and extraretinal signals are combined as needed to guide appropriate responses and that memorized locations are stored relative to the fovea. However, gradual gaze shifts (smooth pursuit or whole-body rotation) and abrupt gaze shifts (saccades) put different constraints on spatial memory processing. Finally, our data suggest that changes in orbital eye position may impact spatial processing independently of changes in the retinotopic position of a memorized target.

Are spatial memories stored in world-, eye-, or head-centered coordinates?

Precisely how retinal and extraretinal signals are combined to maintain spatial memories is not fully understood. In the absence of stable visual landmarks (e.g., in the dark), the brain may combine retinal information with positional information about the subject's gaze direction in the world (Fig. 7A). This combination would yield a stored representation of the stimulus location (M) that is independent of the subject and therefore referenced to the world. The primary advantage of a world-centered representation is that the representation does not need to be altered for movements of the body (including head and eye) that occur after the stimulus has been presented (Fig. 7A, world-fixed trials). A world-centered representation of stimuli that move with gaze, however, must be altered to reflect the movement that these targets undergo whenever gaze moves (Fig. 7A, gaze-fixed trials). The subject reports the remembered location with an eye movement in either case, so the additional processing needed to convert the world-centered representation into an eye-centered saccadic command is the same for both world-fixed and gaze-fixed stimuli (Fig. 7A, right). If we assume that processing additional information adds noise to the stored representation, then memory-guided saccades directed to gaze-fixed locations should be less precise (more variable) than saccades to world-fixed locations after a change in gaze (Fig. 7B). This pattern bears no resemblance to the data that was actually obtained (Fig. 4), allowing us to rule out that a world-centered representation is used to perform this task.

View larger version (25K): [in this window] [in a new window]   Fig. 7. Spatial computations for memory systems operating in world-, eye-, and head-centered and nonlinear coordinates. In each model, a world-fixed target changes position relative to the fovea (T) by an amount that is equal and opposite to the change of eye-in-head (E) plus the change of head-in-body (H) position, which is equal to the change in gaze position (G): T = G. A gaze-fixed target remains at a constant position relative to the fovea: T = 0. Signal flow graphs show how these equations would be implemented in systems operating in each of the three different coordinates (left). First, a memory vector (M arrow) is constructed (left). Next, a target reference frame is selected (switch, set arbitrarily to world-fixed in the figures). Changes in gaze (G, shaded box) are taken into account, and finally a saccadic response is produced (right). The predicted consequences of each architecture on memory saccade variabilility is shown to the right of the graph (right). A: in a model using world-centered coordinates, a spatial memory is formed by adding target and gaze position vectors. Changes in gaze perturb only memories of gaze-fixed targets, leading to higher variability for gaze-fixed trials relative to world-fixed trials of any type (B). C: in a model using eye-centered coordinates, a spatial memory is formed using only the target position vector. Changes in gaze perturb only memories of world-fixed targets, leading to higher variability for world-fixed trials relative to gaze-fixed trials of any type (D). E: in a model using head-centered coordinates, a spatial memory is formed by adding target and eye position vectors. Changes in head position (i.e., rotations) perturb memories of world-fixed targets, while changes in eye position (pursuit and saccades) perturb memories of gaze-fixed targets, leading to higher variability under these conditions (F). G: in a 4th model, the spatial memory is stored as a nonlinear combination of target position and eye position. Like the eye-centered model, changes in gaze perturb the memory of world-fixed targets. In addition, eye movements perturb the memory whether the target is world-fixed or gaze-fixed. If these 2 perturbations (retinotopic updating and eye position updating) introduce noise into the spatial memory independently, the variability should be highest for world-fixed trials with eye movements, lowest for gaze-fixed trials without eye movements (i.e., gaze-fixed rotations), with an approximately equal differential between world- and gaze-fixed trials for each type of shift (H).

Visual signals enter the brain in eye-centered coordinates. It therefore seems plausible to expect spatial information to be stored relative to the eyes. In an eye-centered storage system, only the retinal position of the visual stimulus is stored (Fig. 7C). If the stimulus is world-fixed, both eye-in-head movements and movements of the head in the world (i.e., any type of gaze perturbation) lead to adjustments of the stored signal as they occur. However, if the stimulus is gaze-fixed, these adjustments are unnecessary because the eye-centered position does not change when gaze changes. With an eye-centered storage mechanism, remembering the position of world-fixed stimuli requires the greater number of operations, and memory-guided saccades to world-fixed stimuli would be more variable than saccades to gaze-fixed stimuli (Fig. 7D). Saccade variability after rotational and pursuit gaze shifts are consistent with this pattern (Fig. 4), suggesting that spatial memories used to perform this task are stored relative to the eyes and not relative to the world.

A third type of model suggests that external locations may be processed in a coordinate system referenced to the head or body (e.g., Zee et al. 1976). For the purposes of the present discussion, we will treat only head-referenced systems because in our paradigm the two are equivalent (animals were not free to move their heads relative to their bodies). To produce a head-centered representation, retinal signals are added to eye position signals (Fig. 7E). This form of storage could be useful if a large number of eye movements (but not head movements) are likely to occur prior to the memory-guided response. A head-centered representation of a world-fixed stimulus is adjusted only when the head (or, in our case, the turntable) moves. A gaze-fixed stimulus, conversely, is repositioned within the head-centered representation only when the eyes move in their orbits (i.e., pursuit and saccadic shifts, but not rotations). Our data do not conform to the predictions of the head-centered storage model (Fig. 7F) because saccade variability showed the same pattern after pursuit eye movements and whole-body rotations: world-fixed responses were less precise than gaze-fixed responses. Variability from saccadic shift trials was marginally higher for gaze-fixed stimuli than for world-fixed stimuli, but this difference did not reach statistical significance and will be discussed further in the following text.

A fourth memory model, or class of models, posits a nonlinear or multiplicative combination of retinal signals with eye position signals (Zipser and Andersen 1988). Visual receptive fields are often gain modulated by orbital eye position (Andersen et al. 1990). The fact that the modulation is nonlinear means that changing the retinal position of a stimulus does not have the same effect on the representation as moving the eyes the same amplitude in the opposite direction. Thus while the head-centered signal (discussed in the preceding text) can be maintained by monitoring the difference between the retinal and eye position signals, the nonlinear code requires both signals to be monitored continuously, with changes in either retinal inputs or eye position signals perturbing the representation (Fig. 7G). If these two perturbations (retinotopic updating and eye position updating) introduce noise into the spatial memory independently, a pattern of variability shown in Fig. 7H might be expected. (Note that as the noise in the eye position updating drops to 0, the predictions of the nonlinear model approach those of the eye-centered model.) We observed greater variability after smooth pursuit than after rotation within both gaze- and world-fixed conditions. This finding is consistent with the nonlinear model and suggests that changes in eye position may perturb the stored representation independently of changes in the retinal position of a stimulus.

In summary, based on simple assumptions of noise accumulation resulting from discrete information processing operations in the brain, we have ruled out the possibility that explicit world- or head-centered representations underlie performance in our spatial memory task. The visual system appears to use a representation of the stimulus position that is primarily retinotopic. However, our results support the idea that the representation also relies on eye position signals, requiring adjustment when the eyes move regardless of how the stimulus moves (Li and Anderson 2001).

Visually guided saccades are a special case of gaze perturbation

One might expect that the effect of eye movements on stored signals would not depend on how the eyes have moved. Instead, we found clear differences after fast and slow gaze shifts. While memory-guided saccades that followed rotation or pursuit were more variable under world-fixed compared with gaze-fixed conditions, memory-guided saccades that followed a visually guided saccade were equally variable under world- and gaze-fixed conditions. We found both a decrease in variability on world-fixed trials (relative to world-fixed pursuit and rotation) and an increase in variability on gaze-fixed trials (relative to gaze-fixed rotation; Fig. 4). These findings imply that visually guided saccades may, on the one hand, be easier for the visual system to adjust for, and on the other hand, harder for the system to ignore. Like variability, accuracy was also systematically different for saccades compared with pursuit and rotation (Fig. 5).

Why should accuracy and variability be different after saccadic versus nonsaccadic gaze perturbations? Saccadic signals might have privileged access to spatial processing. Saccades are primarily used to sample the visual scene, and constructing an accurate spatial representation of the scene is an important aspect of this sampling procedure. The visual system may require that the metrics of saccades be more closely monitored than the metrics of other gaze changes.

Other explanations for why saccadic gaze shifts may appear as a special case derive from the metrics of saccades. Saccades are much faster and of shorter duration than the rotation and pursuit movements that we employed. Small errors in the gaze change signal may accumulate over time, resulting in larger errors for more prolonged gaze shifts. To test this possibility, it would be necessary to quantify gaze compensation ability after short, rapid rotations or smooth pursuit. Alternatively, spatial processing may be improved when the amplitude of the gaze change is known from the start of the movement (as in a visually guided saccade), compared with cases in which the amplitude of the gaze change is open-ended and must be continually monitored (as in pursuit or rotation). Also, saccades are more frequent than rotational gaze changes (Flanders et al. 1999) or pursuit movements, and, in contrast to slow gaze changes, almost never involve stimuli that remain fixed with respect to gaze. Any or all of these differences may result in saccades being processed differently than rotations or pursuit.

Potential sources of noise in combining retinal and extraretinal signals

At least two sources of noise could account for the differences observed between world- and gaze-fixed memory-guided saccade variability. First, the internal estimates or representations of rotation and pursuit may themselves be noisy (i.e., estimates of an identical movement may vary from trial to trial). Second, noise may be introduced in the process of combining a retinotopic target location with an estimate of the gaze shift. In a two-dimensional world, this combination would be a vector sum: initial retinotopic location minus gaze shift equals world-fixed location. In a three-dimensional world, the combination of gaze and retinotopic signals is more complex. For example, the effect of a head rotation on the eye-centered location of a world-fixed target depends on the head's rotational axis, provided that the target is not at optical infinity (Medendorp and Crawford 2002; Snyder and King 1992; Viirre et al. 1986). The effects of Donders' law add still more complexity to the conversion of a memory into a motor plan (Klier and Crawford 1998). As a result, the computation required to compensate a world-fixed location for a gaze shift is more complex than a vector summation, and the physical implementation of this computation may add noise into the result.

Our data do not directly distinguish between these sources of noise, but the results obtained with different types of gaze shifts offer some insight. Unlike adjusting for rotation and pursuit, adjusting for a saccade did not introduce any additional noise into the behavioral output. This suggests that the internal estimate of a rapid gaze movements is less variable than the internal estimate of a slow gaze movement. It also suggests that the process of combining the saccade-related change in eye position with the stored spatial memory is a low noise operation (compared with the same operation for rotation and pursuit). Surprisingly, memory variability was higher on gaze-fixed trials when the eyes moved (visually guided saccades and smooth pursuit eye movements) compared with when the eyes remained orbitally fixed (rotations). This difference suggests that even when a subject has learned to ignore the change in gaze position, and thus no combination of retinal and extraretinal signals is necessary, changes in the position of the eyes may still disrupt spatial memory. This disruption suggests that changes in eye position may be registered within the nervous system even when they are not explicitly being used to modify spatial memories.