In everyday life, we efficiently find objects in the world by moving our gaze from one location to another. The efficiency of this process is brought about by ignoring items that are dissimilar to the target and remembering which target-like items have already been examined. We trained two animals on a visual foraging task in which they had to find a reward-loaded target among five task-irrelevant distractors and five potential targets. We found that both animals performed the task efficiently, ignoring the distractors and rarely examining a particular target twice. We recorded the single unit activity of 54 neurons in the lateral intraparietal area (LIP) while the animals performed the task. The responses of the neurons differentiated between targets and distractors throughout the trial. Further, the responses marked off targets that had been fixated by a reduction in activity. This reduction acted like inhibition of return in saliency map models; items that had been fixated would no longer be represented by high enough activity to draw an eye movement. This reduction could also be seen as a correlate of reward expectancy; after a target had been identified as not containing the reward the activity was reduced. Within a trial, responses to the remaining targets did not increase as they became more likely to yield a result, suggesting that only activity related to an event is updated on a moment-by-moment bases. Together, our data show that all the neural activity required to guide efficient search is present in LIP. Because LIP activity is known to correlate with saccade goal selection, we propose that LIP plays a significant role in the guidance of efficient visual search.
People are highly efficient when searching or foraging for an item in a cluttered environment. To do this, subjects need to be able to ignore objects that are dissimilar to the target item and to keep track of which items they have already examined to avoid looking at the same one repeatedly. To allow stimuli similar to the target to be highlighted, it has been proposed that the brain creates a “priority map” of the outside world that incorporates both bottom-up and strong top-down inputs (Serences and Yantis 2006). This theory is based on saliency map models of attentional allocation (Julesz 1984; Koch and Ullman 1985; Treisman and Gelade 1980), in particular the models of Koch, Itti and colleagues (Itti and Koch 2001). Although saliency maps include both bottom-up and top-down inputs, we prefer the use of the term priority map as it removes any ambiguity about the preferential role of salience (bottom-up information) in guiding attention (Fecteau and Munoz 2006; Serences and Yantis 2006). The overall concept is that incoming visual information is prioritized based on salience and then integrated with top-down feedback, such as the suppression of task irrelevant stimuli, modulation due to reward contingencies or prior expectations. Our hypothesis is that covert attention is allocated based on the topography of the map on a moment-by-moment basis, and eye movements are guided to the peak of the map. We believe that the lateral intraparietal area (LIP) of posterior parietal cortex acts as such a map in which features or locations are represented by levels of activity related to the attentional priority at that location (Ipata et al. 2009) and which is used to guide both covert (Bisley and Goldberg 2003; 2006) and overt (Gnadt and Andersen 1988; Ipata et al. 2006a; Roitman and Shadlen 2002; Thomas and Pare 2007) attention.
In this study, we asked whether the activity in LIP is sufficient to guide efficient search by studying responses while monkeys performed a foraging task. Prior studies have shown that LIP activity differentiates between task-relevant targets and task-irrelevant distractors as visual search begins (Balan and Gottlieb 2006; Balan et al. 2008; Ipata et al. 2006a; Thomas and Pare 2007). In this study, we asked whether this differentiation is maintained in ongoing search. If so, then the activity in LIP is sufficient to guide eye movements to target-like stimuli. The second and more novel question we asked was whether LIP activity maintains a representation of stimuli that have been examined. If so, then it would allow for the efficient guidance of search by keeping the eyes away from stimuli that have already been fixated. One potential way of accomplishing this is the process of “inhibition of return,” which is included in saliency map models (Itti and Koch 2000; Koch and Ullman 1985). Based on a psychophysical finding (Klein 2000; Posner and Cohen 1984), the process suppresses the activity representing an item or location once it has been examined (Itti and Koch 2001). Thus we hypothesized that the LIP response to potential targets should be reduced once those items have been fixated.
All experiments were approved by the Chancellor's Animal Research Committee at UCLA as complying with the guidelines established in the Public Health Service Guide for the Care and Use of Laboratory Animals. Two rhesus monkeys (8–10 kg) were implanted with head posts, scleral coils, and recording cylinders during sterile surgery under general anesthesia (Bisley and Goldberg 2006); animals were initially anesthetized with ketamine and xylazine and maintained with isofluorane.
We recorded extracellular single-unit activity from area LIP using tungsten microelectrodes guided by coordinates from magnetic resonance imaging (MRI) images. Recorded neurons were considered to be in LIP if they showed typical LIP responses, such as a visual burst, delayed sustained activity or a peri-saccadic burst, during the memory-guided delayed saccade task (Barash et al. 1991). After calculating the size and position of the receptive field of each neuron using an automated memory-guided delayed saccade task covering 9 or 25 points, the behavioral task was run and neural data were recorded. We discriminated action potentials during the recording epoch using the MEX pattern spike sorter. The experiments were run using the REX system (Hays et al. 1982), and sorted spikes were time stamped and stored at 1 kHz.
Stimuli were presented on a Samsung SyncMaster 1100DF CRT running at 100 Hz. The temporal precision of stimulus onset was captured by a photoprobe on the corner of the monitor. To begin a trial of the foraging task (Fig. 1), the monkeys had to fixate a spot placed on the left, right, or center of the screen; the position was chosen at the beginning of the session and was usually changed once midway through the session. After a delay of 450–700 ms, an array of five potential targets (T) and five distractors (+) were presented. One of the targets had a juice reward associated with it, such that if the monkey looked at it for 500 ms within 8 s after the start of the trial, he would get the reward. As in previous unrestricted viewing visual search tasks (Bichot et al. 2005; Mazer and Gallant 2003), the stimuli were arranged in such a fashion that when the monkey looked at one stimulus, the receptive field of an LIP neuron encompassed another stimulus (black circles and ovals in Fig. 1). The stimuli were always the same size (1.2° × 0.8°) and large enough for the animals to perform the task exceedingly well at all eccentricities tested. On each trial, the spatial arrangement of the stimulus array was identical, but the positions of the targets and distractors were randomly assigned.
To quantify the animals' behavior, we looked at all of the animals' informative saccades and compared their search strategy with an ideal model to elucidate how well the animals were following the basic rules of the task. For this analysis, we considered informative saccades to be those that resulted in the monkey looking at a stimulus for >400 ms. We interpreted fixations lasting <400 ms as an indication that the monkey was aware that the stimulus he was looking at was not the target because fixations of this duration would never get rewarded. In other words, if the animal knew a stimulus would not provide a reward (i.e., it was a target that had been looked at before or was a distractor), he would move on well before 500 ms. These fixations can be seen in the bimodal distribution of fixation durations seen in Fig. 2 A, which shows the distribution of fixation times for the two monkeys. It is worth noting that the informative saccade designation is an operational definition based on the distribution of fixation times. This definition helps to exclude saccades that result in short fixation durations during which the animal is clearly aware that the item he is fixating is not the reward loaded target. Fixations with durations of >500 ms were completely informative; if the monkey was looking at a potential target and did not receive the reward, then he could deduce that the stimulus was not the actual target. We have included fixations with durations between 400 and 500 ms because these are in the tail of the distribution of long saccades, so it is possible that from the monkey's perspective they were informative (even though they were not long enough for him to correctly deduce whether the stimulus he was fixating was the target).
It is possible that saccades to distractors were made under the assumption that the distractor was a target and only after the animal started fixating the distractor did he realize his mistake. However, the fixation durations and distribution shapes were similar to the fixations made at previously fixated targets, so we assumed that these saccades were being made in the same way as saccades to previously fixated stimuli. It is possible that some of the saccades were made to distractors under the assumption that they were targets. This would mean that we overestimated the animal's efficiency, but it also means that we have underestimated the difference in response between targets and distractors by assuming that all distractors are ignored when some may not be. We felt it was more appropriate to underestimate the difference in response than to overestimate it, as would happen if we assumed all saccades to distractors were made under the assumption that the distractor was actually a target.
MODELING EFFICIENCY OF SEARCH.
The model we used to quantify the animals' saccadic behavior is based on the variable memory model of visual search (Arani et al. 1984; Horowitz 2006). This model is aimed at seeing how well the animals followed the basic rules of the task. The original model had three parameters: encoding probability, recall probability, and target identification probability. We removed the encoding probability, as our stimuli were clearly distinguishable but included a parameter that accounted for the fact that not all eye movements we included were completely informative. The model searched through five potential targets and five distractors until it found the target. Each “saccade” was considered informative, so time was not incorporated. The first parameter was the target identification probability. This is a measure of the probability that the observer knew which stimuli were important (the targets) and which were not (the distractors). When the value is 1, the model never looks at a distractor because it has no attentional priority. When this value is zero, the model is equally likely to look at a target or distractor. The second parameter was the recall probability. This is the probability that the model remembered all of the stimuli it had previously examined. When the value is 1, the model remembers all the locations it has checked and always finds the target without looking at any stimulus twice. When this value is zero, the model has no knowledge of what stimuli it has seen and randomly goes to any stimulus (as governed by the target identity probability) other than the one it was already at. We included a fixed parameter that was set by the proportion of noninformative saccades (i.e., fixations with durations between 400 and 500 ms) each monkey made. This parameter resulted in the model not retaining information about the stimulus it was currently looking at on a proportion of saccades set by the proportion of fixations that lasted between 400 and 500 ms.
The values of our two free parameters were obtained by finding the values that produced the best match with the behavioral data. This was performed using a brute force method; we tested all value pairs for the two parameters varying between 0 and 1 with a resolution of 0.02. Each pair of weights was tested with 50,000 simulated trials. The distribution of the number of informative saccades for each weight pair was compared with the actual number of informative saccades made by the individual monkey, and an R2 was calculated using all the points up to and including the last category for which at least one trial had that number of informative saccades. The parameter values that produced the optimal R2 values for each animal were considered the best estimates for the model.
Neural data analysis
Data were recorded from 54 LIP neurons (29 from monkey E and 25 from monkey C). We analyzed neural activity during fixations in which there was a single object inside the receptive field. Data were aligned by either the beginning of fixation or by the beginning of the following saccade. Spike density functions were calculated for visualization only, by convolving spike trains with a Gaussian kernel with a 10-ms sigma. Quantitative analysis was based on spike rates within two windows (gray patches in Figs. 6B and 7B): a 350-ms window starting 150 ms after the end of the last saccade (the “fixation” window) and a 100-ms window starting 50 ms before the beginning of the next saccade (the “peri-saccadic” window). Fixations were only included from trials in which the animal performed the task correctly (>97% of trials) and from fixations in which the eye remained on the stimulus for ≥500 ms. Data were analyzed using custom code written in Matlab (Mathworks).
To examine whether the activity in LIP could explain efficient visual search, we performed extracellular microelectrode recordings from LIP neurons in two rhesus monkeys that were trained on a visual foraging task (Fig. 1), similar to the scanning task used by Sommer (1994). In this task, monkeys searched through five potential targets (T) and five distractors (+) to find the target that was loaded with reward. After the stimuli appeared, the monkeys were free to move their eyes to find the reward-loaded target within 8 s. To get the reward they had to fixate the loaded target for 500 ms, this lead to a strategy in which the monkeys usually looked from target to target, waiting at each for ∼650 ms (Fig. 2).
OVERALL SACCADIC BEHAVIOR.
Monkeys performed the task very well and were usually able to find the rewarded target in >97% of trials per session. With the presence of five targets, an optimal strategy would result in an average of three saccades per trial with ≥500-ms intersaccadic intervals. However, we found that the monkeys made 4.82 saccades per trial. Examination of the data showed that the main reason for the increased number of saccades was that the animals often did not fixate every stimulus they saw for a full 500 ms. This can be seen in Fig. 2A, which shows the distribution of fixation times for the two monkeys. Although the majority of the fixations in both animals lasted >500 ms, there was clearly a bimodal distribution of fixation times. Almost half (44%) of the fixations times of <400 ms were from fixations in which the monkey fixated a distractor (white bars). Both monkeys realized that fixating a distractor would not result in a reward, and so these fixations were usually short; neither monkey fixated a distractor for >400 ms in >0.62% of fixations. Of the remaining short fixations, only 17.4 and 9.4% went to targets in monkeys E and C, respectively. The remaining short saccades were not toward any stimuli on the array. While these saccades represent close to 40% of the short fixations, they only accounted for 8.8 and 14.4% of all saccades in monkey E and C, respectively. Often these were rapid zigzag saccades that took the eyes out of the array and then quickly back onto it.
When uninformative fixations, i.e., those that lasted <400 ms, were removed from the data set, then the mean number of saccades per trial was reduced to 3.08. This was significantly greater than three (P ≪ 0.001, t-test), suggesting that on some occasions the animals looked at the same target twice within a trial. To analyze this behavior, we plotted the fixation times on the first and second fixations of a single target (Fig. 2B). Both panels show three main clusters of data: in the top left; bottom right; and top right regions of the plots. In addition to these clusters, both monkeys show a small proportion of trials in which a target was briefly examined twice within a trial (bottom left region). These mostly represent the very small number of trials in which the monkeys made more than two saccades to a particular target. The cluster on the bottom right is from trials in which the first fixation lasted >500 ms, but the second fixation was brief (<400 ms). This represents trials in which the monkey probably remembered that he had fixated the object and did not wait for the reward period before leaving it. A second, smaller, cluster is from trials in which the first fixation of the target lasted <400 ms and the second lasted for >500 ms. These data represent trials in which the monkey came back to a target that he had fixated and waited the for the appropriate time because he had fixated it for less than the 500 ms the first time. The third cluster is from trials in which both fixations lasted for >500 ms (top right cluster). These data are the most interesting as they imply that the monkey may have forgotten that he had already examined the stimulus for 500 ms. In monkey E, these fixations account for 5.68% of all fixations that lasted >400 ms, in monkey C, they account for only 1.84%. Thus while the animals looked at stimuli twice for >400 ms, they did so rarely.
To see how long it took the monkey to forget whether he had examined a stimulus for 500 ms when he looked at it a second time for >400 ms, we plotted the second time of fixation against a measure of the time since he had fixated it last. Figure 2C shows the raw data, and D shows the means of these data plotted against time for the two monkeys. The — show sigmoidal lines of best fit. Because there is mostly a bimodal distribution of fixation times, the sigmoid fit is really giving an indication of what proportion of current fixations are short and what proportion are long. In doing so, we obtain a metric of the time course of any change. It is clear that the chance of fixating a target that had already been fixated for >500 ms increased as time went on. This increasing probability reached a plateau around 3,000 ms, which correlates to approximately six saccades. However, at this time, there are still a large number of second fixations that last <400 ms. These data suggest that there may be a time course over which the animals start to forget which stimuli they have fixated for 500 ms, but this is not absolute, as second fixations were occasionally short ≤3,000 ms and beyond. We should note that from our data we are unable to differentiate between the possibility that the animals forget which stimuli they fixated and the possibility that they forgot how long they fixated the stimulus the first time they saw it.
SACCADIC BEHAVIOR FROM INFORMATIVE SACCADES.
To quantify animals' efficiency on the foraging task, we looked at all of the animals' informative saccades (i.e., those with fixations >400 ms) and compared their behavior with a model to elucidate how well the animals were following the basic rules of the task. The model incorporated two free variables, target identification probability and recall probability, which are sufficient to completely explain efficient search in this task. Figure 3, A and B, shows the distributions of the number of informative saccades made by each animal within all of their recording sessions. For the vast majority of trials, both animals made five informative saccades or less with an almost equal distribution of trials in each condition of four informative saccades or less. To find a good fit of these distributions using the model, we tested parameter values between 0 and 1 with a resolution of 0.02. The output of the model was compared with the animal's performance and an R2 was calculated. Figure 4 A shows the R2s for the pairs of parameter values ranging from 0.6 to 1 for monkey C. In both monkeys, it became obvious that for almost any value of the recall probability parameter, the best fit was when the target probability value was 1, and the globally highest R2 was always along this line. As such, to identify the best value for the recall probability, we plotted the R2s for all the recall probability values when the target identification probability was 1 (Fig. 4B) and found the peak of the curve. The model outputs showing the number of informative saccades using the best fit parameter values for the two monkeys are shown in Fig. 3, C and D. For monkey E, the best parameter values were 1.0 and 0.92 for the target identification and recall parameters, respectively, and for monkey C, the values were 1.0 and 0.97. The same data are plotted together with the behavioral data in a cumulative fashion in Fig. 3, E and F. It is clear from these graphs that with these parameter values, the models fit the behavioral data exceedingly well. This suggests that both monkeys performed the task with almost perfect efficiency in relationship to distinguishing between a target and distractor and with very high efficiency (>92%) in remembering what items they had fixated before.
It is possible (albeit unlikely) that one way to perform the task without recall would be to have a default search pattern that would be followed on every single trial. For example, the animal may first look down the left column, then up the middle column, then down the right column, until he gets the reward. In this case, the monkey would not have to remember where he has been; he would just make eye movements along the same path on each trial. It seemed to us that this would be unlikely given that the stimulus array was orientated in a different position in each session (to match the LIP receptive field) and that there are 252 different combinations of target positions for each array. However, to test this, we calculated the most visited locations in the first, second, and third saccades of the all trials within each session (which we shall refer to as T1– T3) and asked how often the animal went to these locations in that order when all three locations contained a target and the monkey made three or more saccades. If there was a true stereotyped path in all trials, the monkeys would have started from T1 and proceeded to T2 and T3 in every trial. This behavior would lead to a probability of one in our analysis. The median proportion of trials in which the animals did this was 0.185 (95th percentile session: 0.55), which was significantly different from one (P ≪ 0.001).
The animals did not follow a stereotyped scan path after starting in a different place on each trial. If the animals started by making a saccade to a random target and then followed their stereotyped scan path, the probability that they would look at T1 followed by T2 and then T3 in any trial with at least three saccades (and targets at all 3 locations) is 0.4 (green vertical dashed line in Fig. 5). The black line in Fig. 5 shows the cumulative distribution from the data. Clearly the median (vertical black dashed line) is substantially different from the green data (P≪ 0.001). However, we know that the animals did not start at a random location on each trial; they most often went to T1. To take this into account, we ran a simulation for each session (with 50,000 simulated trials per session), using the probability of the saccade going to T1 based on the actual probability from that session. The remaining first saccades could go to any of the remaining four targets. We then asked what proportion of trials would have the monkey look at T1 followed by T2 and then T3 under the same conditions as before. The data from these simulations are presented in blue in Fig. 5. Given that the monkeys most often started by looking at T1, the probabilities of finding the order of T1, T2, and T3 in a trial is very high and is substantially and significantly higher than in the actual data (P ≪ 0.001).
We should note that the monkeys clearly use some search strategies because patterns of eye movements were repeated. However, because starting locations changed and different patterns of eye movements were often seen when targets were positioned in the same locations, we can be assured that they are not using a default search pattern to perform the task. This is important because if the animal does not use a completely stereotyped search pattern, then it must be using memory to perform the task efficiently.
In the foraging task, LIP neurons responded more to targets than to task irrelevant distractors. Figure 6 B features spike density functions from a single neuron when a target (red trace) or distractor (blue trace) was in the receptive field aligned by the beginning of fixation (left) and by the beginning of the next saccade (right). Even before the saccade had finished bringing the stimulus into the neuron's receptive field (time 0, left), the response of this neuron was stronger when a target was in the receptive field (red trace) than when a distractor was in the receptive field (blue trace). This predictive response is common in LIP (Duhamel et al. 1992) and was significant in 10 of the 54 neurons studied (P < 0.05, t-test on the 50-ms window before the beginning of fixation). More importantly, the response to a target was stronger than the response to a distractor throughout the fixation period. This was also seen at the population level in each monkey individually (Fig. 6, C and D). Interestingly, a closer examination of the population spike density functions shows that immediately following a saccade, the population response produces a small burst of activity that does not clearly differentiate between target or distractor. This is more obvious in monkey E (Fig. 6C) but is still present in monkey C. This burst is similar to the undifferentiated visual response seen in LIP when new stimuli flash on within the receptive field (Bisley et al. 2004; Ipata et al. 2006a; Thomas and Pare 2007) and occurs earlier than the onset of the response seen in area V4 in similar search tasks (Bichot et al. 2005; Mazer and Gallant 2003).
To quantify the difference in response during the period of fixation between saccades, we examined the number of action potentials during a 350-ms window starting 150 ms after the beginning of fixation (gray patch Fig. 6B, left). In the example neuron, the response in this window was significantly stronger for the target than for the distractor (P ≪ 0.0001, t-test). This difference remained up until the end of the fixation and during part of the following saccade (P ≪ 0.001, t-test during gray patch in right panel). Thirty-eight of the 54 neurons showed significantly more activity over the 350-ms epoch (P < 0.05, t-test), and 37 of 54 showed significantly more activity over the 100-ms peri-saccadic period (P < 0.05, t-test). To see the extent of this effect in the neuronal population, we calculated the average activity for all 54 neurons and plotted the responses to the distractors against the responses to the targets in the 350-ms fixation (Fig. 6E) and 100-ms peri-saccadic (Fig. 6F) windows. Consistent with the single neuron spike density functions, the population response was significantly greater to the target than to the distractor in both windows (P ≪ 0.0001, Wilcoxon sign-rank tests) and in the individual animals independently (P < 0.0038, Wilcoxon sign-rank tests). These results show that the activity in LIP differentiates between the targets and task-irrelevant distractors, thus contributing to the efficiency of search by highlighting only potential targets. These results are consistent with previous studies that have shown enhanced activity in LIP to behaviorally relevant stimuli (Colby et al. 1996; Dorris and Glimcher 2004; Gottlieb et al. 1998; Platt and Glimcher 1999; Sugrue et al. 2004) but additionally demonstrate that this information continues within the time scale of a trial and is transferred successfully across multiple saccades.
Although the mean response to a target was significantly greater than to a distractor, we found that the response to a target depended on whether the animal had looked at it previously in the trial or not; neurons tended to respond more to a target that had not been fixated than to a target that had been fixated. Figure 7 B shows the mean spike density functions for the same neuron as in Fig. 6B under conditions in which the target in the receptive field had already been fixated (red) or had not been fixated before in that trial (blue). The response of the neuron was significantly lower when there was a target inside the receptive field that the monkey had already fixated compared with a target that it had not fixated (P ≪ 0.0001). This sustained difference was seen in the population responses of both animals (Fig. 7, C and D) and was significant in 28 of 54 neurons during the 350-ms epoch and 31 of 54 during the 100-ms peri-saccadic period (P < 0.05, t-test). This effect was also significant across the population as a whole (P ≪ 0.0001, Wilcoxon sign-rank tests), in the individual animals (P < 0.004; Fig. 7, E and F) and across saccades within trials (P < 0.002, Wilcoxon sign-rank tests). Although the response to targets that had been fixated was lower than to targets that had not been fixated, it was still higher than the response to a distractor (e.g., black dotted lines in Fig. 7, B–D; P ≪ 0.0001, Wilcoxon sign-rank tests on population data). This result suggests that while foraging, once a stimulus of interest has been looked at, the activity representing it in LIP is reduced.
LIP neurons are known to give an enhanced response when a saccade is planned into the receptive field (Gnadt and Andersen 1988). Consistent with this, we found that the responses to targets that had not been fixated and were the goal of the next saccade were significantly higher than to targets that had not been fixated, but were not the goal of the next saccade (Fig. 8, A and B). Each point in Fig. 8A represents the average activity of a neuron during the 350-ms window in conditions in which targets that had not been fixated were either the goal of next saccade or not. Twenty-five of the 54 neurons showed a significantly higher response to the targets that had not been fixated that were the goal of the next saccade compared with those that were not (P < 0.05, t-test). This effect was magnified when the traditional (Barash et al. 1991) peri-saccadic period was examined (Fig. 8B). In this window, 40 of the 54 neurons had an enhanced response when the next saccade was made toward the target in the receptive field. In both cases, these effects were significant at the population level (P ≪ 0.0001, Wilcoxon sign-rank tests) and in the individual animals (P < 0.0002).
The enhanced responses to targets that have not been fixated cannot be trivially explained by the greater activity in the subset of fixations in which the target in the receptive field will be the goal of the next saccade. It is possible that the elevated activity seen in the mean population response to targets that have not been fixated is entirely due to the inclusion of fixations in which the next saccade will be made to the target in the receptive field. To show that this is not the case, we calculated the average activity in conditions in which the target in the receptive field had been fixated and in which the target in the receptive field had not been fixated but was not the goal of the next saccade (Fig. 8, C and D). We found that in both the 350-ms fixation period and the peri-saccadic period, the population response was significantly less to the target that had been fixated than to the target that had not been fixated even though it was not the goal of the upcoming saccade (P < 0.001, Wilcoxon sign-rank tests). These results were also seen for the two monkeys individually (P < 0.012) and in 19 and 14 of 54 neurons in the fixation and peri-saccadic periods, respectively (P < 0.05, t-test). This means that whether or not a saccade will be made into the receptive field, the response of the neuron differentiates between a target that has been fixated or not.
Role of reward probability in modulating activity
The reduced activity seen in response to targets that have already been fixated is likely due to a reduction in their reward probability. Previous studies have shown that the activity of LIP neurons can be strongly modulated based on the likelihood that the stimulus in the receptive field will lead to a reward (Dorris and Glimcher 2004; Platt and Glimcher 1999; Sugrue et al. 2004). In our task, the monkeys know that the distractors have no reward related to them, so the responses to distractors are lower than to targets. Likewise, one can think of targets that have been looked at for 500 ms as stimuli that have no reward related to them—after looking at it and getting no reward, the monkey can rule it out as a potential target.
The reward probability of the remaining targets did not change after each saccade. Given that reward probability can explain the reduced responses to targets that have been fixated, we asked whether the reward probabilities for all the other targets are dynamically updated throughout the trial. If this was the case, then the responses to targets that had not been fixated should increase as more potential targets are ruled out and the reward probabilities of the remaining targets increases (from 0.2 to 0.25 to 0.33 to 0.5 to 1 after 0, 1, 2, 3, and 4 targets have been fixated, respectively). We found that the response to targets that had not been fixated increased modestly after the first three saccades (Fig. 9). Data are plotted from fixations in which a target that had not been fixated and was not the goal of the next saccade was in the receptive field (Fig. 9, ■) and fixations in which a target that had not been fixated but was the goal of the next saccade was in the receptive field (Fig. 9, ▴). Because the latter data set could include a confounding motor component, we focused on the former data set, but plot both for completion. The responses to targets that had not been fixated increased after each of the first three saccades, albeit not in the almost exponential way expected. Further, the response to the targets that had not been seen but were the goal of the next saccade actually decreased slightly after the fourth saccade. Together, these results suggest that the modulation of activity may not be related to reward probability. To test this, we examined the response to distractors under the same conditions (Xs, Fig. 9). We found that the response to distractors also changed as more items had been examined. Because the reward probability of the distractor does not change (their reward probabilities are 0 because rewards are never associated with them and the monkeys' behavior indicates that they know this), these data suggest that there may be a global increase in activity through these first three fixations. Indeed, when the data were normalized across conditions in which 1–3 targets had been fixated, the changes in response to targets that had not been fixated could be entirely explained by the changes in response to the task-irrelevant distractor (Fig. 9, inset), whose reward probability did not fluctuate during the trial.
We have shown that our two monkeys performed the visual foraging task with a high degree of efficiency. They acted as if they were completely able to differentiate between targets and distractors and as if their recall of items they had already seen was >92%. We have also shown that activity within LIP includes two important mechanisms that can explain the high efficiency in performance. First, as shown previously, LIP incorporates top-down information so that it can differentiate between potential targets and task-irrelevant distractors (Balan and Gottlieb 2006; Balan et al. 2008; Buschman and Miller 2007; Gottlieb et al. 1998; Ipata et al. 2006a; Thomas and Pare 2007). In addition to this, our data have shown that this goal directed activity is retained across multiple saccades within a single trial. Second, LIP keeps track of what stimuli have been examined, marking them off as they are looked at (Dickinson and Zelinsky 2005; Klein 1988; Thomas et al. 2006), by reducing the activity in response to those stimuli on the priority map. Consistent with previous findings, we also found that activity at the goal of the upcoming saccade was greater than the response to an identical stimulus with the same behavioral history that was not the goal of the saccade. These data are consistent with the hypothesis that saccades are made to the peak of activity in LIP (Ipata et al. 2006a; Thomas and Pare 2007). These three effects can best be seen in Fig. 10, which summarizes the neural responses in LIP under all the main stimulus and behavioral conditions.
Together, these data show that activity in LIP is sufficient to explain efficient search and, when combined with the evidence that activity in LIP may guide eye movements (Gnadt and Andersen 1988; Ipata et al. 2006a; Thomas and Pare 2007), suggest that LIP activity may play an active role in the visual search process. This conclusion is consistent with clinical studies that have found that patients with parietal lesions revisit items in cancellation or search tasks (Mannan et al. 2005; Zihl 1995), indicating that the mechanism to keep search efficient, by keeping track of what items they have checked, has been adversely affected.
An interesting feature of this result is that the reduction of activity to a previously fixated stimulus is essentially a form of spatial short-term memory in which less information needs to be stored as time progresses. Rather than making a second list of the potential targets and then marking and remembering which have been fixated, the same map that guides the eye movements can incorporate this memory by suppressing the activity of objects that it deems are no longer potential targets. In this case, spatial memory itself presents much like a form of top-down suppression. While the mechanism that drives this suppression is not known, the most likely possibility is related to reward. Responses in LIP appear to be related to reward probability or expectancy under several behavioral conditions (Dorris and Glimcher 2004; Platt and Glimcher 1999; Sugrue et al. 2004). In our task, when the animal fixates a potential target and is then not rewarded, he can deduce that the item has no reward associated with it. This information would then be incorporated into the priority map activity by reducing the activity representing the reward value of that item. We suggest that this reduced activity does not quite reach the level seen in response to a distractor because of the extensive training the animals have had in ignoring the distractors. This lower response may be indicative of another top-down signal that is driven by overtraining, analogous to that seen when animals are trained to ignore salient popout stimuli (Ipata et al. 2006b).
Although the responses to targets that had been fixated were reduced, we did not find a corresponding increase in response to the remaining targets that had yet to be seen. Given that LIP responses are known to incorporate reward information, we were surprised to see that the responses to the remaining targets did not increase as the probability that they would yield a reward increased. We interpret this result as suggesting that that reward probabilities across the entire visual scene are not updated after each saccade; only stimuli that have been just fixated have their responses modulated. This provides a simple and efficient mechanism for monitoring the world; once information is embedded in the map, it is shifted from neuron to neuron after eye movements by peri-saccadic remapping (Duhamel et al. 1992) and remains relatively stable until an event occurs. Thus the only energy required to change information is the suppression of the stimulus at fixation. Because we did not record the activity from any neurons with receptive fields at the fovea, we cannot ascertain when, during the fixation period, the suppression may occur. Prior studies that have found LIP activity related to reward value, utility, or expectancy (Dorris and Glimcher 2004; Platt and Glimcher 1999; Sugrue et al. 2004) have examined these responses across trials or blocks. We also found a resetting of the map at the beginning of each trial. We suggest that these events act as interrupts and reset the relevant locations on the map. This is also consistent with the updating of decision information over time (Churchland et al. 2008; Leon and Shadlen 2003; Roitman and Shadlen 2002; Yang and Shadlen 2007).
A minor proportion of the neurons we recorded from showed little response differences between conditions; these neurons tended to have low absolute responses to our stimuli. It has been proposed that there may be two groups of neurons in LIP, those that are only spatially selective and those that are tuned for both spatial location and a feature dimension (Ogawa and Komatsu 2009). If we assume that the neurons that play a role in the priority map are the neurons that are only spatially tuned, then we may be underestimating the pooled response on the priority map by including the neurons that are not responsive to our stimuli. This concept is enticing, not only because it appears to enhance our effects but also because it unites the view of LIP as a priority map with studies that have seen nonspatial responses, such as the encoding of directional information (Fanini and Assad 2009), shape information (Sereno and Amador 2006; Sereno and Maunsell 1998), numerical information (Roitman et al. 2007), categorical information (Freedman and Assad 2006), and nonspatial information (Freedman and Assad 2009), in LIP.
The reduced responses we find to stimuli that have been fixated are consistent with the mechanism included in saliency map models of visual search to mark off what locations have already been examined (Itti et al. 1998; Koch and Ullman 1985). The designers of these models included the suppression of locations that had already been attended based on the psychophysical finding of inhibition of return (IOR)—a slowing down of reaction times at a location starting 300–500 ms after attention had been transiently allocated to that location (Klein 2000; Posner and Cohen 1984). This was thought of as a way of preferentially guiding attention to novel locations and is a mechanism that optimizes the allocation of attention, just as we have described optimizing the guidance of eye movements. Because it has been suggested that LIP plays a role in both the allocation of covert attention (Bisley and Goldberg 2003, 2006) and the guidance of saccades (Andersen and Buneo 2002; Gnadt and Andersen 1988; Ipata et al. 2006a; Thomas and Pare 2007), this mechanism provides a way for a single area to help accomplish efficient search under both circumstances. Munoz and colleagues have suggested that reaction times may be a function of the time it takes for new neural activity to reach a threshold, and under conditions in which IOR is seen, the suppression of activity creates a slower neural response and thus slower reaction times (Dorris et al. 2002; Fecteau and Munoz 2005). In this way, suppression of activity in LIP at a preattended location could explain the psychophysical phenomenon of IOR and is consistent with the hypothesis that parietal cortex may play a role in this phenomenon (Vivas et al. 2003).
It is worth noting that while we have focused on LIP, areas such as the frontal eye field (FEF) and the superior colliculus (SC) have also been implicated in the allocation of attention (Cavanaugh and Wurtz 2004; Ignashchenkova et al. 2004; Moore et al. 2003) and, in particular, in visual search (Buschman and Miller 2007; McPeek and Keller 2002; Thompson et al. 1996). Studies in the SC (Dorris et al. 2002; Fecteau and Munoz 2005) and FEF (Bichot and Schall 2002) have also found activity that is consistent with the slowing down of reaction times at previously attended areas but have concluded that these mechanisms are probably driven from other areas. LIP connects directly to both the SC and FEF (Andersen et al. 1990) and thus may drive the modulations reported in these areas.
This work was supported by the Gerald Oppenheimer Family Foundation, the Kirchgessner Foundation, a Klingenstein Fellowship Award in the Neurosciences, an Alfred P. Sloan Foundation Research Fellowship, and National Eye Institute Grant R01 EY019273-01.
We thank the Goldberg lab and J. Fuster, and D. Bernhardt-Walther for comments on a previous version of this manuscript and the members of the UCLA DLAM for superb animal care.
- Copyright © 2009 the American Physiological Society