| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
1Department of Engineering, University of Leicester, Leicester, United Kingdom; 2Computation and Neural Systems, California Institute of Technology, Pasadena; 3Division of Neurosurgery and Semel Institute for Neuroscience and Human Behavior, University of California, Los Angeles, Los Angeles, California; and 4Functional Neurosurgery Unit, Tel Aviv Medical Center and Sackler Faculty of Medicine, Tel Aviv University, Tel Aviv, Israel
Submitted 4 February 2007; accepted in final form 28 July 2007
 |
ABSTRACT |
|---|
|
|
|
INTRODUCTION |
|---|
|
; Desimone et al. 1984; Gross et al. 1969, 1972; Hung et al. 2005; Kiani et al. 2005; Logothetis and Sheinberg 1996; Logothetis et al. 1994; Miyashita 1988; Perrett et al. 1982, 1992; Sato et al. 1980; Schwartz et al. 1983; Tanaka 1996; Young and Yamane 1992). Functional brain imaging of the fusiform gyrus along the ventral pathway (Haxby et al. 2001; Kanwisher et al. 1997) as well as clinical lesion data (Damasio et al. 2000; Farah 1990) support the inference that such neuronal representation is the basis of visual recognition and categorization. These structures project to the medial temporal lobe (MTL), including the hippocampus and the amygdala (Cheng et al. 1997; Saleem and Tanaka 1996; Suzuki 1996). It is from the human MTL that our group has recorded individual neurons responding to pictures of individuals, landmarks, and animals (Fried et al. 1997; Kreiman et al. 2000a,b; Quian Quiroga et al. 2005). About one third of these responsive neurons were selectively activated by completely different views of a given individual or object and in some cases even by letter strings with their names (Quian Quiroga et al. 2005).
How information is represented by a population of neurons can be quantified in an objective manner by inferring the stimulus from its associated firing pattern (Abbott 1994; Abbott et al. 1996; Brown et al. 2004; Keysers et al. 2001; Rieke et al. 1996; Salinas and Abbott 1994; Warland et al. 1997). Such decoding constitutes an objective method to quantify how much stimulus information can be read-out from a neuronal population. We here apply a linear classifier to determine, from the activity of simultaneously recorded neurons in the MTL, which picture was shown to the patient on a trial-by-trial basis. We also determine how these predictions develop in time, how they depend on the number of neurons, and how they depend on correlations among them.
|
|
METHODS |
|---|
|
The data come from 34 sessions in 11 patients with pharmacologically intractable epilepsy (all right-handed, four males, 17 to 49 yr old). Extensive noninvasive monitoring did not yield concordant data corresponding to a single resectable epileptogenic focus. Therefore they were implanted with chronic depth electrodes for 7–10 days to determine the seizure focus for possible surgical resection (Fried et al. 1997). Here we report data from sites in the hippocampus, amygdala, entorhinal cortex, and parahippocampal gyrus. Fifteen of these sessions were previously analyzed for invariance of visual representation (Quian Quiroga et al. 2005). These sessions were also used for decoding in the present study. Moreover, they were used to stress further the invariance results using a decoding approach. For this, we studied whether it was possible to discriminate between the different pictures showing an invariant representation and even predict presentations of pictures that were not seen before by the decoding algorithm. In other words, we tested whether, based on invariance, a decoding algorithm was able to generalize.
All studies conformed to the guidelines of the Medical Institutional Review Board at the University of California at Los Angeles and the Institutional Review Board at Caltech. The electrode locations were based exclusively on clinical criteria and were verified by magnetic resonance imaging (MRI) or by computer tomography coregistered to preoperative MRI. Each electrode probe had a total of nine microwires at its end, eight active recording channels, and one reference. The differential signal from the microwires was amplified using a 64-channel Neuralynx system (Tucson, AZ), filtered between 1 and 9,000 Hz and sampled at 28 kHz. Each recording session lasted about 30 min.
Subjects lay in bed, facing a laptop computer on which pictures were shown. The images covered about 1.5 ° and were displayed six times each in pseudorandom order for 1 s. Images were photos of animals, landmarks, celebrities (partially chosen according to the patients' preferences), and photos of people and places unknown to the patients. More details about the stimulus set are available from Quian Quiroga et al. (2005). The interstimulus interval (ISI) was also randomized with the minimum ISI equal to 1.5 s. To enforce attention to the picture presentations, subjects had to respond after stimulus offset whether the pictures contained a face or something else by pressing the "Y" and "N" keys, respectively. As we will see in the following sections, neuronal responses were very selective and therefore they cannot be explained by the performance of this simple categorization task.
Spike detection and sorting
From the continuous wide-band data, spike detection and sorting was accomplished using a stochastic algorithm (Quian Quiroga et al. 2004). (A Matlab implementation of the algorithm as well as exemplary data are available from www.vis.caltech.edu/
rodri.) After band-pass filtering between 300 and 3,000 Hz, an automatic threshold was set at
 | (1) |
n is an estimate of the standard deviation of the background noise. Note that taking the standard deviation of the signal (including the spikes) could lead to unreliable threshold values, especially in cases with high firing rates and large spike amplitudes. In contrast, by using the estimation based on the median, the interference of the spikes is diminished (Quian Quiroga et al. 2004). We heuristically found the criterion of 5n to be optimal for our data. Although this value is relatively high and it increases the probability of missing low-amplitude spikes, it minimizes the number of false positives (i.e., the detection of noise crossing the threshold by chance). False positives can be easily discriminated from large-amplitude spikes after sorting, but they contaminate multiunit clusters; i.e., those clusters comprising activity from several neurons that could not be split further due to their low signal-to-noise ratio.
Once spikes are detected, the algorithm uses the wavelet transform for extracting features of the spike shapes that are used as inputs to the clustering algorithm. This gives a dimensionality reduction that outperforms results using principal-component analysis (PCA) or the whole spike shape (Quian Quiroga et al. 2004). Clustering—i.e., assigning spikes with similar shapes to the same unit—is done using superparamagnetic clustering, a stochastic method that does not assume any particular distribution of the clusters. Superparamagnetic clustering groups the data into clusters as a function of a single parameter, the temperature, which can be changed by the user if the automatic clustering is not satisfactory. Sometimes, clusters can be chosen from different temperatures and spikes not assigned to any of the clusters can be merged to the nearest cluster.
After sorting, the clusters were classified into single units or multiunits. This was done based on 1) the spike shape and its variance, 2) the ratio between the spike peak value and the noise level, 3) the ISI distribution of each cluster, and 4) the presence of a refractory period for the single units (i.e., <1% spikes within <3-ms ISI).
Data analysis
The response to a picture was defined as the median number of spikes across trials between 300 and 1,000 ms after stimulus onset. Similarly, the baseline for each picture was the median spike count between 1,000 and 300 ms before stimulus onset. Given our relatively low number of trials per picture, we used the median instead of the mean to decrease the effect of outliers, such as a spontaneous burst of several spikes in one of the trials. A unit was considered responsive if the activity to at least one picture fulfilled two criteria (Quian Quiroga et al. 2005): 1) the median number of spikes was larger than the average baseline (across pictures) plus 5 SDs; and 2) the median number of spikes was at least two. Responsive pictures were those that elicited a significant response in at least one unit.
In a first stage, for each session separately we predicted which of the responsive pictures was shown in each trial using only the firing of the responsive neurons, assuming that it would be possible to predict the other pictures if more neurons responding to them had been recorded. Note that the selection of responsive units was done automatically using the above-mentioned criterion and can be seen as the first step of the decoder. In a second stage, for each session we used all simultaneously recorded neurons to predict all the pictures shown to the subject. This was done to verify that our results were not due to a particular definition of responsiveness. Out of an average of 88.4 pictures shown in each session (SD: 11.9, range: 57–114), the average number of responsive pictures per session was 15.9 (SD: 11.2, range: 4–50). Six extra sessions had fewer than three responsive pictures and were not included in the analysis. The number of responsive units was on average 7.8 (SD: 4.5, range: 2–19).
Selectivity measure
Responses of MTL neurons were very selective in the sense that each unit fired to only very few of the pictures shown based on our criterion for responsiveness defined earlier. To rule out that this was not due to the choice of a large threshold for defining responses, we quantified selectivity using a novel index: S. Figure 1 illustrates the procedure with simulated responses. We simulated a neuron with 100 uniformly distributed random firings (Fig. 1A, left); a second simulated neuron was obtained by multiplying 99 of these responses by 1/3 so that only a single response retained its original value (Fig. 1A, right). If we denote by fi the firing of a given neuron to the stimulus i (i = 1, ..., N), we can define the normalized number of "responses" R(T) as the relative number of stimuli with firing larger than a threshold T
 | (2) |
(x) = 1 for x > 0; (x) = 0 for x 
0. Figure 1B shows the normalized number of "responses" R(T) for M = 1,000 threshold values in equal steps between the minimum and the maximum firing (fmin and fmax, respectively). The area under this curve (A) is given by
 | (3) |
 | (4) |
|
. They defined selectivity (or more precisely breadth of tuning) as the ratio
 | (5) |
Decoding
For each session and in separate runs, the numbers of spikes between 300 and 600, 300 and 1,000, and 300 and 2,000 ms for each trial were used as inputs to the decoding algorithm. Trials were represented as points in an m-dimensional space, each coordinate showing the number of spikes for each of the (simultaneously recorded) m responsive units. One at a time, the picture shown in each trial was predicted based on the distribution of all the remaining trials (leave-one-out decoding). The result was averaged by considering each of the six trials left out.
Trials to be decoded were classified using a Fisher linear discriminant algorithm (Duda et al. 2001). To simplify, let us consider the case of two classes to be decoded (i.e., two different pictures) given the firing of m neurons. The first step is a dimensionality reduction by projecting the m-dimensional measurements (i.e., the firing of the m neurons in each trial) onto a line where the samples of each class are optimally separated. The direction of this line (
) is the one that maximizes the ratio of the between-class over the within-class distances. If we denote by 
1 and 2 the centers of the cluster of points corresponding to class 1 and class 2, respectively, the within-class scatter matrix is given by
 | (6) |
)
 | (7) |
The second and final step is to assign the trial to be predicted to one of the two classes, for example, by taking the one that has the minimum Euclidean distance in the direction of . This procedure can be generalized to multiple number of classes (in our case corresponding to the number of images in each session: C), where instead of looking for the optimal separating line, one looks for an optimal (C – 1)-dimensional hyperplane and the trials to be predicted are assigned to the class whose center is the closest in the C – 1 hyperplane. Although in principle the dimensionality reduction performed with Fisher linear discriminants should improve decoding performances, in our case similar results were obtained with a Naïve Bayesian classifier and a Nearest Neighbor classifier (see Supplementary Material).
Decoding results were plotted in the form of "confusion matrices." The values on a given row i and column j of a confusion matrix represent the (normalized) number of times a presentation of picture i is predicted to be picture j. If the decoding is perfect, i = j for all trials and the confusion matrix should have entries equal to one along the diagonal and zero everywhere else. Performance at chance levels should be reflected in a matrix in which each entry has equal probability 1/n, where n represents the number of pictures. Decoding performance was quantified as the percentage of correct predictions, which is the mean of the diagonal of the confusion matrix.
Statistical analysis
For assessing statistical significance of the decoding results, two separate tests were performed. First, we tested whether the difference between the percentage of correct predictions and chance performances (one over the number of pictures) for the different sessions was larger than zero using a t-test. Second, we assessed significance of decoding performances for each session separately. Because the outcomes of the predictions of each picture presentation can be regarded as a sequence of Bernoulli trials, the probability of successes in a sequence of trials follows the binomial distribution (Soong 2004). Given a probability p of getting a hit by chance (p = 1/M, where M is the number of responsive pictures), the probability of getting k hits by chance in n trials is given by
 |
 |
j=kn P(j)]. Information analysis
Information theory offers an alternative quantification of how much information about the stimuli is contained in the firing of the neurons. This is usually done by calculating the mutual information between the stimulus and the neuronal responses
 | (8) |
). If the logarithm is taken in base 2, the information is measured in bits and it specifies how many stimuli M can be encoded by the population of neurons (M = 2I). In our case, we estimated information using the decoding outcomes from the confusion matrix by calculating the mutual information between the actual stimuli and the decoded stimuli (i.e., between the rows and the columns of the confusion matrix). The maximum information that can be extracted is limited by the number of stimuli M and is given by Imax = log2 M. |
|
RESULTS |
|---|
|
|
Figure 3 shows five simultaneously recorded hippocampal units that selectively fired to at least one of the images. In this session, 19 out of a total of 53 simultaneously recorded units (including the five shown in Fig. 3) were responsive, and altogether they fired to 32 out of the 114 pictures viewed by the patient. The firing to all 32 pictures that elicited responses is shown in Figure S5. A common characteristic across all cells was that responses started 300 ms after stimulus onset and were mainly of three types: 1) they occurred between 300 and 600 ms (e.g., picture 58 for unit 3); 2) they lasted up to 1,000 ms (picture 51 for unit 1); and 3) they continued for up to 2,000 ms after stimulus onset, i.e., even after the stimulus was removed from view (picture 20 for unit 3). All five units had very low baseline activities and a sharp increase in their firing after image onset. Based on their spike shape characteristics and interspike-interval distributions, units 3 and 5 were classified as multiunits and units 1, 2, and 4 as putative single units (see METHODS). The single units were nearly silent during baseline (average <0.01 spike/s) and fired with up to 40 spikes/s to only one or two pictures. The multiunits reached similar activation levels, but had a higher baseline activity (0.12 spike/s).
|
Figure 4 illustrates the spike sorting of the channel containing units 1 and 2 of Fig. 3 (the red and green spikes corresponding to clusters 2 and 3, respectively). The top subplot is a 60-s segment of the continuous data and the threshold (Thr) used for spike detection (in red). The leftmost bottom panel shows the projection of the spike shapes onto the first two wavelet coefficients chosen by the algorithm. Note the presence of four clusters, three of which had quite an elongated shape. The remaining bottom panels illustrate the corresponding spike shapes of the sorted units, including the number of events in each cluster (e.g., 592 for cluster 2). There were 13 detected events not assigned to any cluster (not shown). The first (blue) cluster corresponds to a multiunit. Clusters 2, 3, and 4 were identified as single units, although for cluster 2 there were a few spikes with a different spike shape. Cluster 4 contained a total of only 48 spikes and had no significant responses. On the other hand, clusters 1, 2, and 3 had strong responses elicited by a different set of pictures. Responses of clusters 2 and 3 are shown in Fig. 3 (units 1 and 2, respectively) and those of cluster 1 are shown in Figure S5 (unit 6).
|
The responses of Fig. 3 were very selective in the sense that each unit fired to only one to four of the 114 pictures shown, based on our criterion for responsiveness defined earlier. The selectivity index S (see METHODS) for the five units of Fig. 3 is represented in Fig. 5A. On the left, the median number of responses (across 6 trials) for all 114 images is plotted. Clearly, all five units responded to very few pictures. Figure 5A, right displays the relative number of pictures eliciting responses as a function of a variable threshold. For the five units, S had a value >0.9, compatible with a sparse representation.
|
Population decoding
For each recording session, we predicted in each trial which stimulus was seen by the patient using the other five trials to train the decoder (leave-one-out decoding; see METHODS). In a first stage, we predicted the presentation of pictures eliciting responses. Figure 6A shows the decoding performance for the 32 responsive pictures of the session corresponding to Fig. 3 in the form of a confusion matrix (see METHODS). The inputs to the decoding algorithm were the number of spikes between 300 to 1,000 ms of all 19 responsive units for this session. The percentage of hits (mean across the diagonal) was 35.4%, which is significantly better than chance (1 of 32 images, i.e., 3.1%) with P < 10–49 (Bernoulli test; see METHODS). Considering the number of spikes either between 300 and 600 or between 300 and 2,000 ms after image onset gave very similar results (32% in both cases).
|
Dependence on the number of neurons
Next we studied how decoding performance varied with the number of neurons. For this, we calculated the decoding performance using different combinations of k (k = 1, ..., n) out of n neurons. If there were >30 possible combinations, we randomly picked 30 of them. As illustrated in Fig. 6B, the decoding performance for this session increased linearly with the number of neurons. Indeed, a linear fit (thin red line) gave an R2 value of 0.99 (R2 = 1 for a perfect linear fit). A qualitatively similar result was obtained when considering all pictures and all neurons, as shown in Fig. 6C. In this case a linear fit also gave an R2 value of 0.99. For the other recording sessions, the decoding performances using responsive units to predict the presentation of responsive pictures generally increased linearly with the number of units considered. Linear fits yielded R2 values >0.9 for 28 of 34 sessions. Considering all neurons and all images, linear fits yielded R2 values >0.9 for 21 sessions.
Results for all sessions
Figure 7 summarizes the average decoding performances across all sessions. In this case the presentations of the responsive images were predicted using the firing of responsive units in the 300- to 600-, 300- to 1,000-, and 300- to 2,000-ms poststimulus time windows. The horizontal line marks the average chance level. Decoding performances were significantly better than chance with P < 10–12 (t-test) for each of the three time windows analyzed. Considering each session separately, in 31, 33, and 32 of 34 sessions the decoding performance was significantly better than chance for the 300- to 600-, 300- to 1,000, and 300- to 2,000-ms windows, respectively (P < 0.05, Bernoulli test; see METHODS). There was a significant improvement when taking the 300- to 1,000-ms window in comparison with the 300- to 600-ms window (P < 0.05; t-test). In general, predictions using the 300- to 600-ms window were nearly as good as those with the other two larger windows. This is remarkable, considering that in this 300-ms-long interval an average of 4.14 spikes (SD: 4.47; average number of spikes between 300 and 600 ms poststimulation over all responsive units for their corresponding responsive pictures) were sufficient to specify which image was present. Results were similar when considering all neurons and all pictures. In fact, in 33 of 34 sessions the decoding performance was significantly better than chance when considering the 300- to 1,000-ms window (P < 0.05; Bernoulli test; see METHODS). Interestingly, decoding performances were statistically the same when using all units or only the responsive ones to predict the presentations of all pictures (P = 0.69, t-test). This result stresses the fact that nonresponsive units did not carry additional information for decoding.
|
50% for some sessions. We also quantified decoding performances using the mutual information between the actual and the decoded pictures in the confusion matrices (see METHODS). Overall, the average information per session was 1.96 bits (range: 0.76–3.51), slightly more than half the information that could have been recovered (given by the logarithm of the number of pictures). By dividing the total information by the number of responsive neurons in each session, we obtained a mean value (across all sessions) of 0.25 bits per neuron.
Dependence on the time from stimulus onset
Next we studied the time profile of the decoding performance. Because chance levels depend on the number of pictures to be predicted, and each session had a different number of pictures eliciting responses, we used a normalized measure to average results across all sessions. For each session we defined the normalized decoding performance as
 | (9) |
|
To determine whether trial-to-trial correlations between our simultaneously recorded neurons (an average of 7.79 responsive units per session) carried any extra information, we implemented an approach similar to the 
Ishuffle defined by Averbeck and colleagues (2006). For this, we used the same decoding strategy but pseudorandomly permuted the trials corresponding to the same picture presentation, independently for each neuron. That is, the response of, say, the ith unit to the first presentation of some image was matched to the response of the jth unit to the second presentation of the same image and so on. Then we tested whether the original decoding performance was larger than that of each of the 99 shuffled surrogates generated in this way, thus giving a significance level of P < 0.01. Figure 9 shows the original decoding performances and those for the 99 shuffled surrogates, using the number of spikes in the 300- to 1,000-ms time window as inputs to the decoding algorithm. Results using the other two time windows considered (300–600 and 300–2,000 ms) were qualitatively the same. In no case was the original decoding value larger than the values of all surrogates. This was true for all sessions in any of the three time windows considered. Consequently, any trial-to-trial correlations among disparate MTL neurons must have been minor. However, we cannot rule out the presence of correlations carrying relevant information that may be reflected in specific time patterns in the spike trains. The role of synchronization may be more evident if it would be possible to record units with responses to the same pictures, which was rarely the case.
|
In 15 of the 34 sessions, patients saw between three and eight different views of specific individuals or objects (on average 3.95 views of 13.53 individuals per session). From these data, we recently reported the presence of invariant units, in the sense that they fired selectively to different views (including line drawings and letter strings) of the same familiar individual, such as the actress Jennifer Aniston, the actress Julia Roberts, and so on (Quian Quiroga et al. 2005). From the 15 sessions of this study where invariance was tested, we had a total of 41 individuals or objects showing an invariant representation. Taking these different pictures as inputs to the decoding algorithm (one individual at a time), we found that in 8 of 41 cases, the decoding performance was significantly larger than chance with P < 0.05 (Bernoulli test; see METHODS), and only in four cases with P < 0.01. In general, the decoding algorithm could not distinguish between different presentations of the same individual, reinforcing the idea of an invariant representation by MTL neurons.
One compelling aspect of perception is the ability to deal with novel inputs by means of generalization. Humans can easily recognize a familiar person even though he or she may have a new haircut, wear new clothes, be viewed from a different angle, and so on. Given the invariant representation for individuals described earlier, we reasoned that it might be possible to predict pictures of these individuals even if one particular picture had never been seen by the decoder. To test for this, we grouped all but one picture of the individual with an invariant representation as a single class and checked whether the remaining pictures were correctly predicted to be of this class. For example, we had seven pictures of the actress Jennifer Aniston and established whether presentations of picture 1 of her was recognized as belonging to the same image class as pictures 2 to 7 (the same procedure was repeated for picture 2 and so on). Out of the 41 individuals or objects showing invariance, presentations of 21 of them were correctly predicted based on the unit responses to the other pictures of the same person or object (P < 0.05, Bernoulli test; see METHODS).
|
|
DISCUSSION |
|---|
|
). However, this result depends on the definition of what is considered a response and what is not. To avoid the dependence on the selection of a particular threshold to define what is a significant response, in the current study we quantified the degree of selectivity with a new measure: S. Two other measures that are independent of the definition of responsiveness had been proposed earlier (Olshausen and Field 2004; Rolls and Tovee 1995). In the first case, selectivity is assessed from the kurtosis of the distribution of responses. However, this measure is plausible only when the distribution of responses is symmetric, which is not the case for low-firing neurons. Moreover, distributions with different widths can give the same selectivity value (if they have the same kurtosis), something that it is not desirable. In the current study we used the measure proposed by Rolls and Tovee (1995) for comparison. This measure gives a nonlinear dependence with the number of responses and, as a consequence, very different configurations of responses can yield similar selectivity values. On the contrary, the measure we introduced is linear in the number of responses. Moreover, it gives an intuitive graphic representation of the selectivity of the neurons' responses, as shown in Figs. 1 and 5.
Previous decoding approaches studied how relevant information about the stimuli is represented by the activity of a population of neurons, for instance, predicting the position of a rat in its environment from recordings in hippocampus (Brown et al. 1998; Wilson and McNaughton 1993; Zhang et al. 1998), arm movements (Georgopoulos et al. 1986; Musallam et al. 2004; Quian Quiroga et al. 2006; Serruya et al. 2002; Taylor et al. 2002; Wessberg et al. 2000), and saccades (Quian Quiroga et al. 2006; Scherberger et al. 2005) from sensorimotor cortices in monkeys, and image presentations from spiking activity in monkey inferior temporal cortex (Hung et al. 2005).
Here we demonstrated the feasibility of decoding images from the activity of a few responsive neurons in the human MTL. This decoder had several interesting properties. First, it was largely based on an average of 4.47 spikes between 300 and 600 ms after stimulus onset. Second, in general its performance increased linearly with the number of neurons, within the range of the number of units we considered. Third, by using a simple shuffling procedure we established that trial-to-trial correlations in the response strength between the units did not play an important role in our data, in agreement with findings in animals (for reviews see Averbeck et al. 2006; Oram et al. 1998). It remains to be studied whether precise timing mechanisms may be present. Fourth, predictions of which pictures were presented in each trial were significantly better when considering units after spike sorting, compared with those predictions made when taking all detected events of each channel as multiunit (unsorted) activity. Fifth, decoding performances were statistically the same when considering all units or only the responsive ones. This shows that nonresponsive units did not carry any additional information for decoding. Sixth, it was in general not possible to decode which of the pictures with an invariant representation (e.g., different views of a well-known actress) were presented. This result further stresses our previous claims of invariance in MTL neurons (Quian Quiroga et al. 2005) but now using a decoding approach. Finally, also based on the invariant representation given by MTL neurons, the decoder was capable of generalization—i.e., neuronal responses to images of an individual could be used to decode other images of the same individual not previously seen.
A quantification of the decoding results using information theory showed that each responsive neuron carried on average 0.25 bits of information. Values around 0.3–0.5 bits per neuron have been reported in cortical visual areas in monkeys (Optican and Richmond 1987; Rolls et al. 1997). Higher values were found when considering temporal patterns instead of only the average firing (Optican and Richmond 1987). However, we point out that these values were obtained with different stimulus sets, thus having different saturation limits.
In contrast to distributed codes in which information is represented implicitly by the firing activity of large populations of cells (Abbott et al. 1996; Haxby et al. 2001), these data are in agreement with the existence of a sparse, invariant, and explicit representation in MTL, in the sense that the identity of individuals or objects may be represented by a small number of neurons (Koch 2004; Quian Quiroga et al. 2005). This sparse and invariant representation by MTL neurons is reminiscent of Barlow's theory of "cardinal cells" (Barlow 1972) and it is likely to play a key role in the transformation of visual percepts into long-term and abstract memories. This view is also supported by the long latency of the MTL responses. In particular, the peak decoding performance with MTL neurons was considerably later than the one of around 130 ms found in monkey IT (Hung et al. 2005). The possibility of reading-out information from simultaneously recorded neurons in humans is of considerable value for assessing the feasibility and constrains of brain–machine interfaces, so-called neuroprosthetic devices (Andersen et al. 2004; Musallam et al. 2004; Nicolelis 2001; Serruya et al. 2002; Taylor et al. 2002; Wessberg et al. 2000). Our study shows that such decoding is possible in patients, despite the nonoptimal clinical recording conditions, short experimental sessions, and lack of training.
|
|
GRANTS |
|---|
|
|
|
ACKNOWLEDGMENTS |
|---|
|
|
|
FOOTNOTES |
|---|
1 The online version of this article contains supplemental data. 
Address for reprint requests and other correspondence: R. Quian Quiroga, Department of Engineering, University of Leicester, LE1 7RH Leicester, UK (E-mail: rodri{at}vis.caltech.edu)
|
|
REFERENCES |
|---|
|
Abbott LF, Rolls ET, Tovee MJ. Representational capacity of face coding in monkeys. Cereb Cortex 6: 498–505, 1996.
Andersen RA, Budrick JW, Musallam S, Pesaran B, Cham JG. Cognitive neural prosthetics. Trends Cogn Sci 11: 486–493, 2004.
Averbeck BB, Latham PE, Pouget A. Neural correlations, population coding and computation. Nat Rev Neurosci 7: 358–366, 2006.[CrossRef][Web of Science][Medline]
Barlow HB. Single units and sensation: a neuron doctrine for perceptual psychology? Perception 1: 371–394, 1972.[Medline]
Brincat SL, Connor CE. Underlying principles of visual shape selectivity in inferotemporal cortex. Nat Neurosci 7: 880–886, 2004.[CrossRef][Web of Science][Medline]
Brown EN, Frank LM, Tang D, Quirk MC, Wilson MA. A statistical paradigm for neural spike train decoding applied to position prediction from ensemble firing patterns of rat hippocampal place cells. J Neurosci 18: 7411–7425, 1998.
Brown EN, Kass RE, Mitra PP. Multiple neural spike train data analysis: state-of-the-art and future challenges. Nat Neurosci 7: 456–461, 2004.[CrossRef][Web of Science][Medline]
Cheng K, Saleem KS, Tanaka K. Organization of corticostriatal and corticoamygdalar projections arising from the anterior inferotemporal area TE of the macaque monkey: a Phaseolus vulgaris leucoagglutinin study. J Neurosci 17: 7902–7925, 1997.
Cover T, Thomas J. Elements of Information Theory. New York: Wiley, 1991.
Damasio AR, Tranel D, Rizzo M. Disorders of complex visual processing. In: Principles of Behavioral and Cognitive Neurology, edited by Mesulam MM. Oxford, UK: Oxford Univ. Press, 2000, p. 332–372.
Desimone R, Albright TD, Gross CG, Bruce C. Stimulus-selective properties of inferior temporal neurons in the macaque. J Neurosci 4: 2051–2062, 1984.[Abstract]
Duda OH, Hart PE, Stork DG. Pattern Classification (2nd ed.). New York: Wiley, 2001.
Farah MJ. Visual Agnosia. Cambridge, MA: MIT Press, 1990.
Fried I, MacDonald KA, Wilson C. Single neuron activity in human hippocampus and amygdala during recognition of faces and objects. Neuron 18: 753–765, 1997.[CrossRef][Web of Science][Medline]
Georgopoulos AP, Schwartz A, Kettner RE. Neural population coding of movement direction. Science 233: 1416–1419, 1986.
Gross C, Rocha-Miranda C, Brender D. Visual properties of neurons in inferotemporal cortex of the macaque. J Neurophysiol 35: 96–111, 1972.
Gross CG, Bender DB, Rocha-Miranda CE. Visual receptive fields of neurons in inferotemporal cortex of the monkey. Science 166: 1303–1306, 1969.
Haxby JV, Gobbini MI, Furey ML, Ishai A, Schouten JL, Pietrini P. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 293: 2425–2430, 2001.
Hung C, Kreiman G, Poggio T, DiCarlo J. Fast read-out of object information in inferior temporal cortex. Science 310: 863–866, 2005.
Kanwisher N, McDermott J, Chun MM. The fusiform face area: a module in human extrastriate cortex specialized for face perception. J Neurosci 17: 4302–4311, 1997.
Keysers C, Xiao D-K, Földiák P, Perrett DI. The speed of sight. J Cogn Neurosci 13: 1–12, 2001.[CrossRef][Web of Science][Medline]
Kiani R, Hossein E, Tanaka K. Differences in onset latency of macaque inferotemporal neural responses to primate and non-primate faces. J Neurophysiol 94: 1587–1596, 2005.
Koch C. The Quest for Consciousness: A Neurobiological Approach. Englewood, CO: Roberts & Company Publishers, 2004.
Kreiman G, Koch C, Fried I. Category-specific visual responses of single neurons in the human medial temporal lobe. Nat Neurosci 3: 946–953, 2000a.[CrossRef][Web of Science][Medline]
Kreiman G, Koch C, Fried I. Imagery neurons in the human brain. Nature 408: 357–361, 2000b.[CrossRef][Medline]
Logothetis NK, Pauls J, Poggio T. Shape representation in the inferior temporal cortex of monkeys. Curr Biol 4: 401–414, 1994.[CrossRef][Web of Science][Medline]
Logothetis NK, Sheinberg DL. Visual object recognition. Annu Rev Neurosci 19: 577–621, 1996.[CrossRef][Web of Science][Medline]
Mishashita Y. Neuronal correlate of visual associative long-term memory in the primate temporal cortex. Nature 335: 817–820, 1988.[CrossRef][Medline]
Musallam S, Corneil BD, Greger B, Scherberger H, Andersen RA. Cognitive control signals for neural prosthetics. Science 305: 258–262, 2004.
Nicolelis MAL. Actions from thoughts. Nature 409: 403–407, 2001.[CrossRef][Medline]
Olshausen BA, Field DJ. Sparse coding of sensory inputs. Curr Opin Neurobiol 14: 481–487, 2004.[CrossRef][Web of Science][Medline]
Optican LM, Richmond BJ. Temporal encoding of two-dimensional patterns by single units in primate inferior temporal cortex. III. Information theoretic analysis. J Neurophysiol 57: 162–178, 1987.
Oram MW, Földiák P, Perrett DI, Sengpiel F. The "ideal homunculus": decoding neuronal population signals. Trends Neurosci 21: 259–265, 1998.[CrossRef][Web of Science][Medline]
Perrett DI, Hietanen JK, Oram MW, Benson PJ. Organization and functions of cells responsive to faces in the temporal cortex. Philos Trans R Soc Lond B Biol Sci 335: 23–30, 1992.
Perrett DI, Rolls E, Caan W. Visual neurons responsive to faces in the monkey temporal cortex. Exp Brain Res 47: 329–342, 1982.[Web of Science][Medline]
Quian Quiroga R, Nadasdy Z, Ben-Shaul Y. Unsupervised spike detection and sorting with wavelets and super-paramagnetic clustering. Neural Comput 16: 1661–1687, 2004.[CrossRef][Web of Science][Medline]
Quian Quiroga R, Reddy L, Kreiman G, Koch C, Fried I. Invariant visual representation by single neurons in the human brain. Nature 435: 1102–1107, 2005.[CrossRef][Medline]
Quian Quiroga R, Snyder L, Batista A, Cui H, Andersen R. Movement intention is better predicted than attention in the posterior parietal cortex. J Neurosci 26: 3615–3620, 2006.
Rieke F, Warland D, de Ruyter van Steveninck R, Bialek W. Spikes: Exploring the Neural Code. Cambridge, MA: MIT Press, 1996.
Rolls ET, Tovee M. Sparseness of the neuronal representation of stimuli in the primate temporal visual cortex. J Neurophysiol 73: 713–726, 1995.
Rolls ET, Treves A, Tovee MJ. The representational capacity of the distributed encoding of information provided by populations of neurons in primate temporal visual cortex. Exp Brain Res 114: 149–162, 1997.[CrossRef][Web of Science][Medline]
Saleem KS, Tanaka K. Divergent projections from the anterior inferotemporal area TE to the perirhinal and entorhinal cortices in the macaque monkey. J Neurosci 16: 4757–4775, 1996.
Salinas E, Abbott LF. Vector reconstruction from firing rates. J Comp Neurosci 1: 89–107, 1994.[CrossRef][Medline]
Sato T, Kawamura T, Iwai E. Responsiveness of inferotemporal single units to visual pattern stimuli in monkeys performing discrimination. Exp Brain Res 38: 313–319, 1980.[Web of Science][Medline]
Scherberger H, Jarvis MR, Andersen RA. Cortical local field potential encodes movement intentions in the posterior parietal cortex. Neuron 46: 347–354, 2005.[CrossRef][Web of Science][Medline]
Schwartz EL, Desimone R, Albright TD, Gross CG. Shape recognition and inferior temporal neurons. Proc Natl Acad Sci USA 80: 5776–5778, 1983.
Serruya MD, Hatsopoulos NG, Paninski L, Fellows MR, Donoghue JP. Brain-machine interface: instant neural control of a movement signal. Nature 416: 141–142, 2002.[CrossRef][Medline]
Soong TT. Fundamentals of Probability and Statistics for Engineers. Chichester, UK: Wiley, 2004.
Suzuki WA. Neuroanatomy of the monkey entorhinal, perirhinal and parahippocampal cortices: organization of cortical inputs and interconnections with amygdala and striatum. Semin Neurosci 8: 3–12, 1996.[CrossRef]
Tanaka K. Neuronal mechanisms of object recognition. Science 262: 685–688, 1993.
Tanaka K. Inferotemporal cortex and object vision. Annu Rev Neurosci 19: 109–139, 1996.[CrossRef][Web of Science][Medline]
Taylor DM, Tillery SIH, Schwartz AB. Direct cortical control of 3D neuroprosthetic devices. Science 296: 1829–1832, 2002.
Warland DK, Reinagel P, Meister M. Decoding visual information from a population of retinal ganglion cells. J Neurophysiol 78: 2336–2350, 1997.
Wessberg J, Stambaugh CR, Kralik JD, Beck PD, Laubach M, Chapin JK, Kim J, Biggs J, Srinivasan MA, Nicolelis MAL. Real-time prediction of hand trajectory by ensembles of cortical neurons in primates. Nature 408: 361–365, 2000.[CrossRef][Medline]
Wilson MA, McNaughton BL. Dynamics of the hippocampal ensemble code for space. Nature 261: 1055–1057, 1993.
Young MP, Yamane S. Sparse population coding of faces in the inferior temporal cortex. Science 256: 1327–1331, 1992.
Zhang K, Ginzburg I, McNaughton BL, Sejnowski TJ. Interpreting neuronal population activity by reconstruction: unified framework with application to hippocampal place cells. J Neurophysiol 79: 1017–1044, 1998.
This article has been cited by other articles:
|
|
 |
|
  F. Mormann, S. Kornblith, R. Q. Quiroga, A. Kraskov, M. Cerf, I. Fried, and C. Koch Latency and Selectivity of Single Neurons Indicate Hierarchical Processing in the Human Medial Temporal Lobe J. Neurosci., September 3, 2008; 28(36): 8865 - 8872. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 R. Q. Quiroga, R. Mukamel, E. A. Isham, R. Malach, and I. Fried Human single-neuron responses at the threshold of conscious recognition PNAS, March 4, 2008; 105(9): 3599 - 3604. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
| Visit Other APS Journals Online |
TOP
ABSTRACT
INTRODUCTION
