| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
1Centre for Neuroscience Studies and Department of Psychology, Queen's University, Kingston, Ontario, Canada; and 2Section for Physiology, Department of Integrative Medical Biology, Umeå University, Umeå, Sweden
Submitted 30 April 2008; accepted in final form 13 July 2008
 |
ABSTRACT |
|---|
|
|
|
INTRODUCTION |
|---|
|
; Carlton 1981; Land et al. 1999; Paillard 1996; Sarlegna et al. 2004; Saunders and Knill 2004). In addition, proprioceptive and/or motor signals related to gaze position can be used to guide the hand; even when the hand is not visible, directing gaze to the target improves reaching accuracy (Prablanc and Martin 1992; Prablanc et al. 1979, 1986, 2003).
In daily tasks, we often reach to previously seen, out-of-view objects, such as when grasping our coffee cup behind our morning newspaper. However, it is not known how people naturally use gaze when reaching to remembered target locations. For several reasons, we might expect people to look at remembered target locations when reaching to them. Such gaze shifts might increase reach accuracy because they would enable us to exploit well-practiced sensorimotor transformations that produce motor commands driving the hand to the fixation point (Henriques et al. 2003). Fixating the remembered target may also simplify the computations required to specify the required arm motor commands (Beurze et al. 2006; Prado et al. 2005). In addition, an accurate gaze shift to the remember target location might allow effective use of afferent and efferent signals related to gaze position to guide the hand. If vision of the hand is available, this strategy might also permit optimum use of visual feedback to control the hand (Paillard 1996; Saunders and Knill 2004).
The aim of this study was to examine, for the first time, where people choose to look when reaching to remembered target locations and, in particular, whether they generate saccades to fixated these locations. To begin to tackle this question, we designed a task in which participants used a pen to mark the remembered locations of four targets presented on a screen under normal lighted conditions. We used multiple targets because manual tasks typically involve a sequence of actions directed to different targets (Ballard et al. 1992; Flanagan and Johansson 2003; Hayhoe and Ballard 2005; Johansson et al. 2001; Land et al. 1999). We used lighted conditions because some visual references are usually present in natural tasks in which people reach to remembered target location.
In the main encoding mode, targets were presented in peripheral vision while participants fixed a central cross, and in the main recall mode, participants were free to use gaze as they wished when marking the remembered target locations. In natural tasks, future target locations may be viewed in peripheral vision but may also be looked at and/or contacted by the hand. To examine the importance of target directed gaze fixations and hand movements during encoding, we included three additional encoding modes in which participants marked the visible targets: 1) using the pen with free gaze, 2) using gaze fixations without hand movement, or 3) using the pen with central gaze fixation. We also examined two additional recall modes in which participants marked remembered target locations: 1) using the pen with central gaze fixation or 2) with gaze fixations only. By examining recall with the pen with central gaze fixation, we could assess whether recall accuracy deteriorates if eye movements were prevented. By examining recall with gaze fixations only, we could assess the capacity of the oculomotor system to access stored representations of target locations. In all 12 combinations of encoding and recall, participants could see the screen and their hand throughout the task. To examine if our findings from the pen-marking task might generalize to other tasks involving reaching to no-longer-visible targets, we included a second experiment in which we examined gaze behavior when participants performed an object manipulation task in near complete darkness after viewing the involved objects. This second experiment complements the first experiment in which visual references were available during recall.
|
|
METHODS |
|---|
|
Five men and four women between 20 and 34 yr of age participated in the main experiment involving the pen-marking task. An additional four women and five men between 22 and 52 yr of age participated in a second experiment involving an object manipulation task. All participants provided informed consent, and both experiments were conducted according to the declaration of Helsinki. None of the participants required or wore corrective lenses or had a history of ophthalmologic or neurological disease. We will first describe the methods for our main experiment and briefly outline the methods for the second experiment, most of which have been previously described (Johansson et al. 2001).
Apparatus
The apparatus and the general procedure of the pen marking task have been previously described (Terao et al. 2002). Targets were presented on a computer screen (28 x 21 cm) aligned in a frontal plane 45 cm ahead of the participant's eyes and could be marked with the tapered tip of a pen (15 cm long and 1.3 cm diam with a 6-cm conical tip). The scene was lit by two compact fluorescent lights (450 lumens; color temperature 2700K) located 
1 m above and 0.5 m on each side of the screen. The three-dimensional position of the tip of the pen was recorded at 60 Hz with an accuracy of ±0.2 cm using a miniature electromagnetic position-angle sensor (FASTRAK, Polhemus, Colchester, VT) attached to the proximal end of the pen. An infrared video-based eye tracker (RK-726PCI pupil/corneal tracking system, ISCAN, Burlington, MA) recorded the gaze position of the right eye in the plane of the screen at 120 Hz. The eye tracker was mounted on a wooden support, and a headrest and bite bar stabilized the head. The participant was seated. We estimated that our system measures gaze in the horizontal and vertical with accuracies of 0.50 and 0.52° of visual angle, respectively (Johansson et al. 2001).
Procedure
Each trial began with a tone (1 kHz for 300 ms) followed by the presentation of a white cross (0.6 x 0.6 cm; 0.8 x 0.8°) at the center of the screen that participants were required to fixate. When the gaze had stayed within 2° of the center of the cross for 1 s, the four targets (filled white circles of diameter 0.3 cm or 0.4°) were presented simultaneously at unpredictable locations against a black background. To ensure that the targets were distributed widely and unpredictably, they were presented at randomly selected angles at four different eccentricities from the cross (1.9, 3.1, 4.4, and 5.7 cm or 2.4, 4.1, 5.5, and 7.2° visual angle from the central cross; see left panels in Fig. 1). Either the targets were presented for 0.25 or 6 s. Four seconds after the targets were extinguished, a second auditory tone (1 kHz for 150 ms) signaled the start of the recall period during which the participants marked the remembered locations of each target on the screen. After six seconds, a third tone (1 kHz for 300 ms) signaled the end of the trial, and the central cross was extinguished. A single trial lasted for either 11.25 or 17 s. The intertrial interval was 2 s.
|
The 12 different combinations of encoding and recall modes were performed in blocks of trials with the order of blocks counterbalanced across participants. For the three blocks involving the cross fixation encoding mode, participants completed 16 trials, 8 for each target presentation duration. In all remaining blocks, participants completed eight trials. The entire experiment, including calibration and breaks between test blocks, lasted 2.5 h. At the start of the experiment, we instructed participants to mark the four target positions, or remembered target positions, as precisely as possible; speed of movement was not stressed in the instructions. When marking target locations with the pen, participants were instructed to lift the pen from the screen between consecutive markings and not to slide the pen over the screen. Before each block, participants were instructed about the required eye and hand movements in the encoding and recall phases. In encoding and recall modes involving pen marking with free gaze, participants were told that they were free to look anywhere and that they did not need to fixate the central cross.
Analysis
The onset and end of each gaze fixation were defined as previously described (Johansson et al. 2001). We defined the gaze fixation associated with pen marking of a specific target as the fixation that was located closest to the marked site within a time window starting 0.4 s before and ending 0.4 s after the pen contacted the screen. The time between pen-markings of targets with free gaze was 0.79 ± 0.23 (SD) s (data from all markings by all participants pooled) and did not differ significantly between marking of remembered and visible targets locations (F1,8 = 2.8; P = 0.13; repeated-measures ANOVA based on median values for each participant). The times at which the pen contacted and lifted off the screen were determined from sharp peaks in pen tip acceleration normal to the screen. For markings performed both during encoding and recall, we determined which of the four marked locations that corresponded to the four targets by computing the sum of squared distances between the targets and marked locations for all 24 possible alignments and selecting the alignment that gave the smallest value. During gaze marking trials, participants sometimes shifted the gaze to more than four locations after the start of the recall period because they refixated a visible or remembered target location. In such cases, we used the first four fixations recorded rather than the four most accurate fixations so as not to bias our results.
To analyze hand-eye coordination, we cross-correlated pen and gaze positions, projected on to the plane of the screen, between the first and fourth target markings with the pen. The pen and gaze position signals were low-pass filtered at 4 Hz, and to determine the gaze and hand lag, we searched for the maximum correlation while shifting gaze position ±0.5 s relative to the pen position in 5-ms steps. To simultaneously cross-correlate horizontal (x) and vertical (y) position signals, we interleaved the x and y gaze positions and the x and y pen positions and, when time shifting these interleaved signals, always shifted by a multiple of two samples.
Unless otherwise indicated, we used repeated-measures ANOVAs based on median values for each participant to assess significant effects of modes of encoding and recall on variables specified in RESULTS. P < 0.05 was considered statistically significant.
Object manipulation task
The second experiment, in which participants performed an object manipulation task in near complete darkness after viewing the objects, was run at the same time as the experiments reported in Johansson et al. (2001) and using the same basic task and apparatus. While seated behind a table, participants used the tips of the right index finger and thumb to grasp, lift, and move a bar (2 x 2 x 8 cm) in a frontal plane, termed the work plane, located 39 cm in front of the participant's eyes. An electronic shutter (Speedglas, Hörnell International, Gagnef, Sweden) located 8 cm in from of the eyes could be used to block completely the view of the scene at any time. When the shutter was closed, the participant was in darkness. Gaze was recorded and calibrated using the same apparatus and procedure described above. Using sensors in the object and attached to the nails, we recorded the three-dimensional position and orientation of the bar and the tips of the right index finger and thumb.
At the start of each trial, the shutter was opened for 3 s while the participant viewed the support surface and bar, a target located on a stand, and, in some conditions, an obstacle in the direct movement path attached to the stand (see Fig. 7). The shutter then closed and, after a 2- to 4-s random delay, a brief auditory tone signaled to the participant to perform the manipulation task. This involved reaching for and grasping the right end of the bar and moving it such that its left end contacted the target. After contacting the target, the participant replaced the bar on the support surface. Participants first performed four consecutive trials without an obstacle and then four trials with an obstacle. The participants had previously performed corresponding trials under lighted conditions (Johansson et al. 2001). To assess performance in this task, we measured the time from hand movement onset to target contact and the height of the tip of the bar when it first arrived within 0.2 cm of the target (in the horizontal). We used the SD of the height of the tip of the bar (computed for each participant) as a measure of variability across trials in reaching performance. In obstacle conditions, we also assessed the obstacle clearance defined as the maximum horizontal distance between the tip of the bar and the obstacle.
|
|
|
RESULTS |
|---|
|
Encoding during cross fixation and recall using the pen with free gaze
While fixating a central cross on a computer screen, participants were presented with four targets in extrafoveal vision for either 0.25 or 6 s in different blocks of trials (encoding phase). The targets disappeared and, after a 4-s delay, participants were required to mark the remembered locations of the targets as accurately as possible with the tip of a hand-held pen (recall phase). Participants could mark the targets in any order and were free to use gaze as they wished. They could also view the screen and their hand at all times. We combined the data from these two target presentation durations because the duration had no effect on gaze or hand behavior during recall (see also Terao et al. 2002).
When marking remembered target locations, gaze behavior was erratic. In some trials, a participant could generate coordinated gaze and hand movements where the amplitude of the gaze movement approached the amplitude of pen movement (Fig. 1A). However, in other trials, the same participant could generate minute gaze movements keeping gaze near the center of the screen (Fig. 1B). In comparison, when participants used the pen to mark visible targets with natural use of gaze (during the pen with free gaze encoding condition), participants showed concurrent gaze and pen movements to each successive target with gaze arriving at the target before the pen (Fig. 1C).
Spatial distribution of gaze fixations and pen markings
During recall with free gaze, the pen tended to undershoot the two most eccentric target locations and overshoot the least eccentric. The left panel of Fig. 2A shows the locations of all pen markings by all participants with reference to target eccentricity. A repeated-measures ANOVA showed that target eccentricity affected pen eccentricity error defined as the distance from the central cross to the marked position minus the distance from the central cross to the target (F3,24 = 14.1; P < 0.0001). As shown in the right panel of Fig. 2A, the majority of gaze fixations during recall were substantially closer to the central cross. Nevertheless, the eccentricity of gaze fixations increased slightly with target eccentricity, and a repeated-measures ANOVA showed that this effect was significant (F3,24 = 5.0; P < 0.01). In contrast, when marking visible targets, the eccentricity of both pen marking and gaze fixations closely matched the eccentricity of the targets (Fig. 2B).
|
The pattern of results shown in Fig. 2 was highly consistent across participants. Accordingly, the distributions of absolute pen marking errors (distance between pen marking and target location) and absolute gaze fixation errors (distance between gaze fixation and target location) were highly consistent across participants when marking both remembered (Fig. 3A) and visible (Fig. 3B) targets. As shown by the purple curves in Fig. 3A, all participants exhibited a wide spread of absolute gaze fixation errors when marking remembered targets. Separate analyses of pen marking and gaze fixation errors in the horizontal and vertical components during recall showed that the horizontal pen marking error (0.68 ± 0.47 cm) tended to be slightly greater than the vertical error (0.53 ± 0.10 cm; mean ± SD based on median values for each participant; t(8) = 3.46; P < 0.01). As may be discerned from Fig. 2A, the error pattern for the remembered targets tended to be horizontally elongated. The horizontal component of the gaze fixation error (1.79 ± 0.26 cm) did not differ significantly from its vertical component (1.69 ± 0.23 cm; t(8) = 1.79; P = 0.11).
|
Spatiotemporal coordination between hand and gaze movements
To examine the spatiotemporal coordination between hand and gaze movements during recall with free gaze, we cross-correlated pen and gaze position signals from each trial to assess the maximum correlation, the lag at maximum correlation, and the gain; that is, the slope of the relationship between hand and gaze position at the maximum correlation lag (see METHODS). For comparison, we also assessed trials in which the same participants marked visible targets. Figure 4A shows maximum coefficients of correlation between pen and gaze movements when marking remembered (red), and visible (black), targets using box plots. Separate box plots are shown for each participant. Figure 4, B and C, shows corresponding box plots for gaze leads and gaze:hand gains.
|
When marking visible targets, the gaze:hand gain was close to unity for all participants (Fig. 4C, black boxes), indicating that gaze and pen moved to similar positions. In contrast, when marking remembered targets, for each participant the gain was far lower and more variable (Fig. 3F, red boxes). Thus the size of the gaze movements when marking remembered targets was smaller and more variable than when marking visible targets. A repeated-measures ANOVA confirmed that the gaze:hand gain was less when marking remembered targets than when marking visible targets (F1,8 = 124.9; P < 0.0001). Importantly, when marking remembered targets, absolute pen-marking error did not correlate with maximum correlation coefficient, gaze lead, or gaze:hand gain (P > 0.05 in all 3 cases; correlations based on data pooled across participants).
Effects on recall of encoding modes engaging hand and eye movements
That fact that participants did not reliably fixate remembered target locations during recall indicates that gaze is controlled differently when reaching to remembered versus visible targets. The question arises whether this would still be the case if participants during the encoding phase fixated the visible targets. If participants could associate each visible target location with an oculomotor command or a proprioceptive signal related to eye position, participants might be able to more accuracy align gaze with remembered target locations during recall. If so, they might be able to exploit effectively gaze- dependent mechanisms similar to those used when reaching to visible targets. Hence, participants might choose to fixate remembered targets under this encoding condition. We also asked whether encoding using the pen with free gaze would prompt fixations of remembered targets during recall. In this encoding mode, participants look at the visible targets, and thus target locations could be associated with efferent and afferent signals related to both eye and arm movements. Finally, we also included an encoding mode in which the participants marked the targets with the pen while fixating the central cross. This enabled us to examine effects on recall behavior of hand movements alone during encoding.
Figure 5A shows distribution of pen markings and associated gaze fixations during recall for the four different modes of encoding. All marking and fixations from all participants are shown. As in Fig. 2C, the pen markings and gaze fixations associated with different target eccentricities have been rotated around the central cross such that targets of increasing eccentricity were located at 0, 90, 180, and 270°. Overall, participants showed similar pen and gaze behavior during recall across the four encoding modes. Figure 5B shows the average absolute pen marking error (black columns; based on participant medians) and gaze fixation errors (purple columns) during recall for each encoding mode. A repeated-measures ANOVA failed to show a difference in pen marking error between encoding modes during recall (F3,24 = 1.06; P = 0.21). In all encoding modes, all participants exhibited erratic gaze behavior and seldom foveated the remembered target locations during recall. This resulted in large gaze fixation errors in all encoding modes that were far greater than gaze fixation errors during encoding using the pen with free gaze (Fig. 5B, white inset columns). A repeated-measures ANOVA indicated nevertheless that the encoding mode affected the absolute gaze fixation error in recall (F3,24 = 4.69; P < 0.01). When participants fixated the visible target during encoding (i.e., in the gaze marking and pen with free gaze encoding modes), the eye movements during recall were slightly larger in amplitude and more accurate. Consistent with this observation, repeated-measures ANOVAs showed that encoding mode affected the maximum correlation (F3,24 = 6.2; P < 0.005; Fig. 4C) and gain (F3,24 = 7.57; P < 0.001; Fig. 4E) of the cross-correlations between pen and gaze position signals. Encoding mode did not affect the gaze lead (F3,24 = 0.68; P = 0.56).
|
Effect of gaze movements during recall on pointing accuracy
The finding that, during recall, pen marking error does not depend on gaze fixation error suggests that hand accuracy is unaffected by gaze direction and that the eye movements observed during recall are not important for guiding the hand. If this is true, pen marking performance should not be affected if eye movements are prevented during recall. To test this prediction, we included a recall mode where we instructed the participants to maintain central fixation while marking the remembered target locations with the pen.
Figure 6A shows the distributions of pen markings and associated gaze fixations during recall with central cross fixation and during recall with free gaze. All marking and fixations from all participants and all four encoding modes are shown. We combined data from all encoding modes because encoding mode had little effect on recall behavior. The pen markings and gaze fixations associated with different target eccentricities have been rotated as in Fig. 2C. Similar pen marking behavior was observed in the two recall modes. The black columns in Fig. 6C show the average absolute pen marking errors (based on participant medians) when participants fixated the cross during recall compared with when they were free to move gaze as they wished. A repeated-measures ANOVA failed to show a different in pen marking error between the two recall modes (F1,8 = 0.22; P = 0.66). The white columns inside the black columns show the absolute pen marking errors when participants marked visible targets during encoding modes that matched the indicated recall modes.
|
As with pen marking, gaze marking tended to undershoot the two most eccentric target locations and overshoot the least eccentric. Target eccentricity influenced the eccentricity error of gaze markings defined as the distance from the central cross to the marked position minus the distance from the central cross to the target (F3,24 = 5.0; P < 0.01). This suggests that similar memory representations were used when marking with the hand and gaze.
Sequential order of target marking during encoding and recall
In conditions where participants marked targets during both encoding and recall, we asked if the order in which targets were marked was preserved from encoding to recall. We found that participants used the same marking order in 70% of trials and that this percentage did not depend on whether the eye or hand was used to mark visible targets during encoding or remembered targets during recall. A repeated-measure ANOVA failed to show main effects of encoding mode (F2,16 = 0.10; P = 0.91), recall mode (F2,16 = 1.48; P = 0.26), or an interaction (F4,32 = 1.45; P = 0.24). To assess possible benefits of preserving marking order, for each combination of encoding and recall modes, we compared marking accuracy, during recall, in trials in which targets were marked in the same order and those in which they were marked in a different order. For the gaze marking, we used absolute gaze fixation errors and, for the two other recall modes, we used absolute pen marking errors. No significant difference in accuracy was observed in any of the nine combinations (3 encoding modes x 3 recall modes), even without correcting for multiple comparisons (P > 0.05 in all cases). Finally, we found no relationship between target marking order (1–4) and either pen (F3,24 = 2.28; P = 0.10) or gaze (F3,24 = 0.68; P = 0.60) marking accuracy during recall when factoring out the effect of target eccentricity described above. Taken together, these results provide little evidence that participants exploited memory systems related to the sequence of targets marked.
Object manipulation task
In our pen marking task, participants could continuously view the hand and pen as well as fixed objects in the scene. In addition, they did not receive sensory feedback informing them whether or not they successfully contacted the remembered target locations during recall. The question arises as to whether similar gaze behavior would be observed under different conditions. To begin to address this question, we analyzed a set of previously collected data in which participants performed an object manipulation task in complete darkness after viewing the task environment. In this manipulation task, participants receive tactile feedback indicating if target objects are successfully contacted, as in most natural tasks involving actions on remembered target locations.
In our object manipulation task, participants used their right hand to reach for and grasp the right end of a bar, move the bar in a frontal plane such that its left end contacted a target, and replace the bar on a support surface (Fig. 7). Participant performed the task in complete darkness after viewing the objects and task environment for 3 s and waiting an additional 2–4 s for an auditory go signal. This task was performed both with and without an obstacle located between the bar and the target (Fig. 7A). We compared gaze behavior when participants performed this task in the dark and with visible objects. We have previously thoroughly described gaze behavior in the latter case (Johansson et al. 2001).
Figure 7A shows the spatial distributions of gaze fixations, recorded up until target contact, when performing the task with visible objects (left column) and in the dark (right column). Each panel shows all fixations from all participants (9) and trials (4). To show the spatiotemporal coordination between gaze and hand movement, these fixations are color-coded depending the phase of the hand movement. These phases are depicted in Fig. 7B, which shows paths of the tip of index finger and left edge of the bar for four single trials from one participant in the obstacle condition. For fixations associated with each phase, we have drawn an ellipse centered at the centroid. The major and minor axes of the ellipse represent the SD along the principle components of variation. Three phases are shown: the reach and grasp phase between the onset of hand movement and onset of bar movement (red dots and curves), the lift phase between the end of the reach and grasp phase and the moment the left tip of the bar approached within 3 cm of the middle of the right edge of the target (green dots and curves), and the target phase between the end of the lift phase and the moment when the left tip of the bar contacted the target. To assign each gaze fixation to a hand movement phase, we determined the phase at the mid-point of each fixation (i.e., the time half way between fixation onset and offset).
As we have shown previously, when the task was performed in the light, gaze fixations were always directed to the grasp point on the bar and the target and were often directed to the obstacle when present. Gaze arrived at each of these landmarks ahead of the hand (or bar in hand). Gaze departed the grasp point and target at around the time of hand or bar contact and departed the obstacle at around the time the tip bar approached closest to the protruding point on the obstacle. Gaze could also be shift from the grasp site to the tip of the bar during the reach and grasp phase. In contrast, when the task was performed in the dark, the gaze fixations associated with all phases of the movement were broadly scattered and were rarely directed to the landmarks fixated when performing in the light. Nevertheless, as can be appreciated form the ellipses shown in Fig. 7A, a weak link between gaze fixation locations and the phase of the movement was observed. For example, gaze fixations associated with the reach and grasp phase tended to be located closer to the bar than gaze fixations associated with the lift and target phases. Finally, participants made approximate twice as many fixations when performing in the light (mean = 8.0 ± 1.2) compared with dark (mean = 4.3 ± 1.21). When performing in the light, small amplitude gaze shifts are often observed while participants directed their gaze to a spatial landmark (Johansson et al. 2001).
All participants were able to perform successfully the manipulation task in the dark. However, movements were slower and more variable when performing in the dark compared with the light. A two-way (obstacle condition x light condition) repeated-measures ANOVA showed that the time from hand movement onset to target contact was greater (F1,8 = 17.1; P < 0.005) when performing in the dark (mean = 4.2 ± 1.0 s) than in the light (mean = 3.0 ± 0.5 s). No effect of obstacle condition was observed. In addition, in the obstacle condition, the maximum horizontal distance between the tip of the bar and the obstacle was greater (F1,8 = 6.4; P < 0.05) when performing in the dark (mean = 5.1 ± 2.0 cm) than when performing in the light (mean = 3.5 ± 0.8 cm). Finally, the variability in the height of the tip of the bar when it first arrives within 0.2 cm of the target (in the horizontal) was greater when performing in the dark (mean = 0.69 ± 0.15 cm) than in the light (mean = 0.22 ± 0.08 cm; F1,8 = 48.4; P < 0.001). Consequently, participant initially missed the target on about a third of the trials when performing in the dark (Fig. 7B, right). Finally, we asked whether reach errors in the dark were related to where participants directed their gaze with reference to the target. We found no significant correlation between the distance between the target center and gaze position (defined as the point where the line of gaze would have interested the work plane) and the distance between the target center and the height of the bar tip when it arrived within 0.2 cm of the target in the horizontal (r = –0.11, P = 0.37, n = 72; data pooled across obstacle conditions and participants). For each trial we used the gaze fixation that was located closest to the target at any time during the lift and target phases.
|
|
DISCUSSION |
|---|
|
Although participants sometimes directed their gaze close to a remembered target location, this did not improve manual accuracy. This finding is in agreement with two studies examining reaches to the remembered location of a peripherally presented target with and without an instructed gaze shift to the remembered location before reaching (Enright 1995; van Donkelaar and Staub 2000). Shifting gaze reduced reach amplitude (Enright 1995; van Donkelaar and Staub 2000) and trial-to-trial variability (Enright 1995) but did not systematically improve reach accuracy across target directions (Enright 1995) and amplitudes (van Donkelaar and Staub 2000). Experiments addressing position-perception errors indicated that the distance to a memorized target location is also underestimated in the presence of visual references when gaze shifts are allowed (van der Heijden et al. 1999).
When reaching to visible targets, gaze behavior clearly influences reach accuracy. Fixating the visible target enables optimal use of sensed and/or predicted gaze position signals (Prablanc and Martin 1992; Prablanc et al. 1979, 1986, 2003) and visual feedback of hand movements referenced to the foveated target for on-line movement adjustments (Berkinblit et al. 1995; Carlton 1981; Land et al. 1999; Paillard 1996; Sarlegna et al. 2004; Saunders and Knill 2004). Our results suggest that different strategies are used when reaching to remembered targets under the conditions we examined. One possible reason why our participants did not fixate remembered target locations is that gaze accuracy cannot be ensured via visual feedback of the target. However, we found that participants could quite accurately direct their gaze to remembered target when explicitly instructed to do so. Therefore from an optimization perspective (Sober and Sabes 2005; van Beers et al. 2002; Vaziri et al. 2006), we might have expected to see gaze shifts to remembered targets so that estimates of target location, derived from gaze direction, could be combined with other estimates to obtain a more accurate estimate of target location for guiding the hand.
Numerous studies have investigated the coordinates in which remembered targets are stored. Studies using single targets presented in dark or near dark conditions have shown that targets can be represented in gaze-centered coordinates (Beurze et al. 2006; Crawford et al. 2004; Engel et al. 2002; Henriques et al. 1998; Pouget et al. 2002; Vaziri et al. 2006). Other studies, in which clearly visible objects are present, have shown that remembered target locations can be represented relative to these objects; that is, in allocentric coordinates (Carrozzo et al. 2002; Dassonville et al. 1995; Diedrichsen et al. 2004; Obhi and Goodale 2005). Many studies suggest that, in general, the brain use multiple frames of reference, including both egocentric and allocentric frames, to represent remembered targets (Andersen and Buneo 2002; Battaglia-Mayer et al. 2003; Bridgeman et al. 1997; Burgess et al. 2002; Burnod et al. 1999; Crawford et al. 2004; Graziano and Gross 1998; Milner and Goodale 1995; Olson 2003) and that these are weighted differently depending on the time course and conditions of the task (Carrozzo et al. 2002; de Grave et al. 2004; Diedrichsen et al. 2004; McIntyre et al. 1997, 1998; Obhi and Goodale 2005). Although, this study was not designed to test the coordinate frame(s) in which targets were encoded, we assume that, in our pen-marking task, participants relied strongly on allocentric cues. However, that gaze was not directed to the remembered targets does not exclude that gaze signals, in general, could have been used. That is, we cannot exclude the possibility that targets were encoded in gaze-centered coordinates and that gaze position, however erratic, was taken into account when recalling target positions.
In our object manipulation task, allocentric cues (i.e., visual references) that could be exploited when recalling target locations were removed. Thus to perform this task, participants presumably relied on target locations represented in egocentric coordinates, possibly including gaze-centered coordinates. The fact that participants did not direct their gaze to remembered targets when performing the manipulation task in the dark suggests that looking at these locations does not facilitate the extraction of target information stored in these egocentric coordinates.
It has been suggested that remembered target locations encoded in gaze-centered coordinates might gradually degrade as successive saccades are generated after the targets have been removed (Karn et al. 1997). This putative degradation could arise from accumulation of errors related to the remapping of target locations required after every saccade (Gnadt et al. 1991; Niemeier et al. 2003; Vaziri et al. 2006). If this is true, and assuming that remembered target locations were encoded in gaze-centered coordinates, we might have expected to see a degradation of recall marking accuracy with successive marking in the pen marking task because participants generally made eye movements while marking. In fact, we might have expected participants to prevent gaze shifts during recall so as to avoiding remapping. We did not observe an increase in marking error with successive markings nor did our participants prevent gaze shifts. However, it should be emphasized that the evidence for saccade-related degradation of remembered target locations is equivocal (Karn et al. 1997). Moreover, because strong allocentric cues were available in our pen marking task, any degradation caused by remapping might be minimal.
We found that recall performance did not improve when hand and/or eye movements were used in encoding. This suggests that, in these conditions, participants did not exploit sensorimotor memories of hand and eye positions, stored when marking visible targets, when recalling the remembered target locations. Moreover, the fact that the four targets were often marked in different orders during encoding and recall (Terao et al. 2002)—even when the same effector (eye or hand) was used—indicates that recall was not based on memorized sensorimotor contingences (Noton and Stark 1971).
We observed a weak and noisy coupling between hand and gaze movements when participants marked remembered target locations with natural use of gaze. These eye movements may be automatically triggered as each target in memory becomes salient in the course of the sequential recall task being performed by the hand. However, in the absence of a visible target, the saccade parameters may not be fully specified, resulting in erratic eye movements. Another alternative is that the eye movements emanate from action schemas implemented for guiding the hand to external locations. When we direct actions toward visible targets, we activate action plans that typically include instructions for task-specific eye movements that provide visual information critical for the task (Flanagan and Johansson 2003; Land and Furneaux 1997). The action plan implemented when marking remembered targets may also call for such eye movements. Again, erratic eye movements could result from the absence of visible targets. Many parts of the neural networks underlying the control of hand and eye movements are strongly interconnected and populations of neurons in a number of structures involved in the control of gaze are tuned by hand movement signals and visa versa (Andersen and Buneo 2002; Battaglia-Mayer et al. 2003; Burnod et al. 1999). This connectivity might lead to "signal leakage" from hand to gaze controllers, resulting in eye movements weakly correlated with hand movement even when task specific gaze movements are not specified. The notion of signal leakage or cross-talk between eye and hand controllers has been invoked to explain other types of interactions between hand and eye movements (Bekkering et al. 1995; Tipper et al. 2001). The brain may allow these loosely coupled eye movements to take place to avoid possible costs associated with actively suppressing eye movements (Prado et al. 2005).
Although eye and hand movements are closely linked in many manual tasks, there are also numerous situations in which they are decoupled and where the hand and eyes are effectively involved in different simultaneous tasks (Hayhoe and Ballard 2005). For example, during an interesting conversation at a dinner party, we may keep eye contact while the hand reaches for and grasps things at remembered locations on the table. Thus a possible advantage of encoding target locations in a way that does not require looking at the targets during recall is that gaze will be available for use in other tasks when reaching to, and acting on, no-longer-visible targets. Thus gaze could be used elsewhere without incurring a cost in manual performance.
|
|
GRANTS |
|---|
|
|
|
ACKNOWLEDGMENTS |
|---|
|
Present address for Yasuo Terao: Department of Neurology, Division of Neuroscience, Graduate School of Medicine, University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8655, Japan.
|
|
FOOTNOTES |
|---|
Address for reprint requests and other correspondence: R. S. Johansson, Section for Physiology, Dept. of Integrative Medical Biology, Umeå Univ., SE-90187 Umeå, Sweden (E-mail: roland.s.johansson{at}physiol.umu.se)
|
|
REFERENCES |
|---|
|
Ballard DH, Hayhoe MM, Li F, Whitehead SD. Hand-eye coordination during sequential tasks. Philos Trans R Soc Lond B Biol Sci 337: 331–338, 1992.[Web of Science][Medline]
Battaglia-Mayer A, Caminiti R, Lacquaniti F, Zago M. Multiple levels of representation of reaching in the parieto-frontal network. Cereb Cortex 13: 1009–1022, 2003.
Bekkering H, Adam JJ, van den Aarssen A, Kingma H, Whiting HT. Interference between saccadic eye and goal-directed hand movements. Exp Brain Res 106: 475–484, 1995.[Web of Science][Medline]
Berkinblit MB, Fookson OI, Smetanin B, Adamovich SV, Poizner H. The interaction of visual and proprioceptive inputs in pointing to actual and remembered targets. Exp Brain Res 107: 326–330, 1995.[Web of Science][Medline]
Beurze SM, Van Pelt S, Medendorp WP. Behavioral reference frames for planning human reaching movements. J Neurophysiol 96: 352–362, 2006.
Bridgeman B, Peery S, Anand S. Interaction of cognitive and sensorimotor maps of visual space. Percept Psychophys 59: 456–469, 1997.[Web of Science][Medline]
Burgess N, Maguire EA, O'Keefe J. The human hippocampus and spatial and episodic memory. Neuron 35: 625–641, 2002.[CrossRef][Web of Science][Medline]
Burnod Y, Baraduc P, Battaglia-Mayer A, Guigon E, Koechlin E, Ferraina S, Lacquaniti F, Caminiti R. Parieto-frontal coding of reaching: an integrated framework. Exp Brain Res 129: 325–346, 1999.[CrossRef][Web of Science][Medline]
Carlton LG. Visual information: the control of aiming movements. Q J Exp Psychol 33A: 87–93, 1981.[Web of Science]
Carrozzo M, Stratta F, McIntyre J, Lacquaniti F. Cognitive allocentric representations of visual space shape pointing errors. Exp Brain Res 147: 426–436, 2002.[CrossRef][Web of Science][Medline]
Crawford JD, Medendorp WP, Marotta JJ. Spatial transformations for eye-hand coordination. J Neurophysiol 92: 10–19, 2004.
Dassonville P, Schlag J, Schlag Rey M. The use of egocentric and exocentric location cues in saccadic programming. Vision Res 35: 2191–2199, 1995.[CrossRef][Web of Science][Medline]
de Grave DD, Brenner E, Smeets JB. Illusions as a tool to study the coding of pointing movements. Exp Brain Res 155: 56–62, 2004.[CrossRef][Web of Science][Medline]
Diedrichsen J, Werner S, Schmidt T, Trommershäuser J. Immediate spatial distortions of pointing movements induced by visual landmarks. Percept Psychophys 66: 89–103, 2004.[Web of Science][Medline]
Engel KC, Flanders M, Soechting JF. Oculocentric frames of reference for limb movement. Arch Ital Biol 140: 211–219, 2002.[Web of Science][Medline]
Enright JT. The non-visual impact of eye orientation on eye-hand coordination. Vision Res 35: 1611–1618, 1995.[CrossRef][Web of Science][Medline]
Flanagan JR, Johansson RS. Action plans used in action observation. Nature 424: 769–771, 2003.[CrossRef][Web of Science][Medline]
Gnadt JW, Bracewell RM, Andersen RA. Sensorimotor transformation during eye movements to remembered visual targets. Vision Res 31: 693–715, 1991.[CrossRef][Web of Science][Medline]
Graziano MS, Gross CG. Spatial maps for the control of movement. Curr Opin Neurobiol 8: 195–201, 1998.[CrossRef][Web of Science][Medline]
Hayhoe M, Ballard D. Eye movements in natural behavior. Trends Cogn Sci 9: 188–194, 2005.[CrossRef][Web of Science][Medline]
Henriques DY, Klier EM, Smith MA, Lowy D, Crawford JD. Gaze-centered remapping of remembered visual space in an open-loop pointing task. J Neurosci 18: 1583–1594, 1998.
Henriques DY, Medendorp WP, Gielen CC, Crawford JD. Geometric computations underlying eye-hand coordination: orientations of the two eyes and the head. Exp Brain Res 152: 70–78, 2003.[CrossRef][Web of Science][Medline]
Johansson RS, Westling G, Bäckström A, Flanagan JR. Eye-hand coordination in object manipulation. J Neurosci 21: 6917–6932, 2001.
Karn KS, Moller P, Hayhoe MM. Reference frames in saccadic targeting. Exp Brain Res 115: 267–282, 1997.[CrossRef][Web of Science][Medline]
Land M, Mennie N, Rusted J. The roles of vision and eye movements in the control of activities of daily living. Perception 28: 1311–1328, 1999.[CrossRef][Web of Science][Medline]
Land MF, Furneaux S. The knowledge base of the oculomotor system. Philos Trans R Soc Lond B Biol Sci 352: 1231–1239, 1997.
McIntyre J, Stratta F, Lacquaniti F. Viewer-centered frame of reference for pointing to memorized targets in three-dimensional space. J Neurophysiol 78: 1601–1618, 1997.
McIntyre J, Stratta F, Lacquaniti F. Short-term memory for reaching to visual targets: psychophysical evidence for body-centered reference frames. J Neurosci 18: 8423–8435, 1998.
Milner AD, Goodale MA. The Visual Brain in Action. Oxford, UK: Oxford, 1995.
Niemeier M, Crawford JD, Tweed DB. Optimal transsaccadic integration explains distorted spatial perception. Nature 422: 76–80, 2003.[CrossRef][Web of Science][Medline]
Noton D, Stark L. Eye movements and visual perception. Sci Am 224: 35–43, 1971.[Medline]
Obhi SS, Goodale MA. The effects of landmarks on the performance of delayed and real-time pointing movements. Exp Brain Res 167: 335–344, 2005.[CrossRef][Web of Science][Medline]
Olson CR. Brain representation of object-centered space in monkeys and humans. Annu Rev Neurosci 26: 331–354, 2003.[CrossRef][Web of Science][Medline]
Paillard J. Fast and slow feedback loops for the visual correction of spatial errors in a pointing task: a reappraisal. Can J Physiol Pharmacol 74: 401–417, 1996.[CrossRef][Web of Science][Medline]
Pouget A, Ducom JC, Torri J, Bavelier D. Multisensory spatial representations in eye-centered coordinates for reaching. Cognition 83: B1–B11, 2002.[CrossRef][Web of Science][Medline]
Prablanc C, Desmurget M, Gréa H. Neural control of on-line guidance of hand reaching movements. Prog Brain Res 142: 155–170, 2003.[Web of Science][Medline]
Prablanc C, Echallier JF, Komilis E, Jeannerod M. Optimal response of eye and hand motor systems in pointing at a visual target. I. Spatio-temporal characteristics of eye and hand movements and their relationships when varying the amount of visual information. Biol Cybern 35: 113–124, 1979.[CrossRef][Web of Science][Medline]
Prablanc C, Martin O. Automatic control during hand reaching at undetected two-dimensional target displacements. J Neurophysiol 67: 455–469, 1992.
Prablanc C, Pélisson D, Goodale MA. Visual control of reaching movements without vision of the limb. I. Role of retinal feedback of target position in guiding the hand. Exp Brain Res 62: 293–302, 1986.[Web of Science][Medline]
Prado J, Clavagnier S, Otzenberger H, Scheiber C, Kennedy H, Perenin MT. Two cortical systems for reaching in central and peripheral vision. Neuron 48: 849–858, 2005.[CrossRef][Web of Science][Medline]
Sarlegna F, Blouin J, Vercher JL, Bresciani JP, Bourdin C, Gauthier GM. Online control of the direction of rapid reaching movements. Exp Brain Res 157: 468–471, 2004.[Web of Science][Medline]
Saunders JA, Knill DC. Visual feedback control of hand movements. J Neurosci 24: 3223–3234, 2004.
Sober SJ, Sabes PN. Flexible strategies for sensory integration during motor planning. Nat Neurosci 8: 490–497, 2005.[Web of Science][Medline]
Terao T, Andersson NE, Flanagan JR, Johansson RS. Engagement of gaze in capturing targets for future sequential manual actions. J Neurophysiol 88: 1716–1725, 2002.
Tipper SP, Howard LA, Paul MA. Reaching affects saccade trajectories. Exp Brain Res 136: 241–249, 2001.[CrossRef][Web of Science][Medline]
van Beers RJ, Baraduc P, Wolpert DM. Role of uncertainty in sensorimotor control. Philos Trans R Soc Lond B Biol Sci 357: 1137–1145, 2002.
van der Heijden AH, van der Geest JN, de Leeuw F, Krikke K, Müsseler J. Sources of position-perception error for small isolated targets. Psychol Res 62: 20–35, 1999.[CrossRef][Web of Science][Medline]
van Donkelaar P, Staub J. Eye-hand coordination to visual versus remembered targets. Exp Brain Res 133: 414–418, 2000.[CrossRef][Web of Science][Medline]
Vaziri S, Diedrichsen J, Shadmehr R. Why does the brain predict sensory consequences of oculomotor commands? Optimal integration of the predicted and the actual sensory feedback. J Neurosci 26: 4188–4197, 2006.
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
| Visit Other APS Journals Online |
TOP
ABSTRACT
INTRODUCTION
) were presented at 4 eccentricities shown by the 4 circles in the left panel (not seen by the participant). The black-gray and blue traces in the left panel represent the positions of the tip of the pen and gaze, respectively, in the plane of the screen. The blue numbers refer to successive gaze fixations, and the matching black numbers denote the segment of pen-tip trajectory (alternating black and gray) from the onset of the fixation until the start of the subsequent fixation. The inset in B shows the gaze trace magnified by 300%. The right panels show pen-tip and gaze positions as a function of time for the trials shown in the left panels. The x- and-y positions are defined in the left graph and the z-position refers to the distance between the pen and the screen where 0 (gray vertical bars) corresponds to screen contact. The small white boxes on the bars indicate target positions in x and y.
