HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Abstract Full Text (PDF) All Versions of this Article: 97/4/3036    most recent 01211.2006v1 Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in ISI Web of Science Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via ISI Web of Science (4) Citing Articles via Google Scholar Google Scholar Articles by Bray, S. Articles by O'Doherty, J. Search for Related Content PubMed PubMed Citation Articles by Bray, S. Articles by O'Doherty, J.

Neural Coding of Reward-Prediction Error Signals During Classical Conditioning With Attractive Faces

¹Computation and Neural Systems and ²Division of Humanities and Social Sciences, California Institute of Technology, Pasadena, California

Submitted 16 November 2006; accepted in final form 11 February 2007

	ABSTRACT

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION GRANTS ACKNOWLEDGMENTS REFERENCES

 
Attractive faces can be considered to be a form of visual reward. Previous imaging studies have reported activity in reward structures including orbitofrontal cortex and nucleus accumbens during presentation of attractive faces. Given that these stimuli appear to act as rewards, we set out to explore whether it was possible to establish conditioning in human subjects by pairing presentation of arbitrary affectively neutral stimuli with subsequent presentation of attractive and unattractive faces. Furthermore, we scanned human subjects with functional magnetic resonance imaging (fMRI) while they underwent this conditioning procedure to determine whether a reward-prediction error signal is engaged during learning with attractive faces as is known to be the case for learning with other types of reward such as juice and money. Subjects showed changes in behavioral ratings to the conditioned stimuli (CS) when comparing post- to preconditioning evaluations, notably for those CSs paired with attractive female faces. We used a simple Rescorla-Wagner learning model to generate a reward-prediction error signal and entered this into a regression analysis with the fMRI data. We found significant prediction error-related activity in the ventral striatum during conditioning with attractive compared with unattractive faces. These findings suggest that an arbitrary stimulus can acquire conditioned value by being paired with pleasant visual stimuli just as with other types of reward such as money or juice. This learning process elicits a reward-prediction error signal in a main target structure of dopamine neurons: the ventral striatum. The findings we describe here may provide insights into the neural mechanisms tapped into by advertisers seeking to influence behavioral preferences by repeatedly exposing consumers to simple associations between products and rewarding visual stimuli such as pretty faces.

	INTRODUCTION

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION GRANTS ACKNOWLEDGMENTS REFERENCES

 
Faces convey a wealth of information and are perhaps the most important visual stimuli for humans in social environments (Adolphs 2001). The attractiveness of a face is a feature that we can perceive quite automatically (Olson and Marshuetz 2005), can subsequently motivate our behavior in terms of mate choice (Rhodes et al. 2005), and bias our beliefs about others' personality and expected success in life (Dion et al. 1972). The effect of attractiveness on human behavior has been documented in the workplace, where it has been shown that attractive individuals enjoy higher salaries (Hamermesh and Biddle 1994) and better employment prospects (Dipboye et al. 1977). These observations have led to the suggestion that preference for facial attractiveness may have evolved to enhance reproductive success (Rhodes 2006; Thornhill and Gangestad 1999).

Recent evidence indicates that attractive faces may act as a form of visual reinforcer as human subjects are prepared to work to gain access to them (Aharon et al. 2001). Although much is now known about the neural circuitry involved in processing the perceptual (Haxby et al. 2000; Ishai et al. 2005; Kanwisher et al. 1997) and affective aspects of facial stimuli (Adolphs 2001; Blair 2003; Whalen et al. 1998), the neural substrates of facial attractiveness are much less well understood. Nevertheless some preliminary studies investigating processing of facial attractiveness have implicated brain regions known to be involved in reward processing, such as the orbitofrontal cortex and ventral striatum (Aharon et al. 2001; Kranz and Ishai 2006; McClure et al. 2003; O'Doherty et al. 2003a,b; Rolls 2000).

Here we aim to address the manner in which facial attractiveness can influence one important aspect of human behavior: behavioral preference. Attractive faces have long been used in advertising as a means of modulating behavioral preferences for specific products. Indeed marketing research has shown that people will evaluate products more favorably when they are presented alongside physically attractive models (Baker and Churchill 1977; Smith and Engel 1968). One possible mechanism for this preference modulation effect is through classical conditioning, whereby an arbitrary neutral stimulus acquires affective value through repeated pairing with a stimulus that has preestablished value such as an attractive face.

In this study, we set out to elucidate the neural mechanisms of this phenomenon by scanning human subjects with functional magnetic resonance imaging (fMRI) while they learned an association between arbitrary affectively neutral visual stimuli (fractals) and attractive and unattractive male and female faces. Before and after the conditioning procedure, we took ratings of pleasantness and preference for the arbitrary fractal stimuli to establish whether behavioral preferences for these stimuli could be modulated as a function of conditioning with attractive faces.

We aimed to characterize the neural processes underlying learning of these preference associations. Modern learning theories propose that such reward-dependent learning is driven by the degree of surprise or unpredictability of a rewarding outcome, or more specifically, errors in predictions of reward (Rescorla and Wagner 1972). Electrophysiological studies in non-human primates implicate the phasic firing of midbrain dopaminergic neurons in encoding reward-prediction errors (Schultz et al. 1997). fMRI studies of human learning have found evidence of reward-prediction error-related activity in known projection sites of dopaminergic cells, especially the ventral striatum, during learning with other forms of natural and abstract rewards such as juice or money (McClure et al. 2003; O'Doherty et al. 2004; Tobler et al. 2006). Given that attractive faces can also be considered as a form of reward, we hypothesized that learning with attractive faces would also engage brain structures known to be involved in reward-prediction error coding such as the ventral striatum.

	METHODS

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION GRANTS ACKNOWLEDGMENTS REFERENCES

 
Subjects

Twenty-eight subjects participated in this study (15 females and 13 males), ranging in age from 18 to 27 [mean: 20.8 ± 2.24 (SD) yr]. All subjects gave informed consent for the study, which was approved by the local research ethics committee. Due to technical difficulties (for 1 subject the experiment stopped during the study due to a software problem, and for 2 other subjects, part of the data set was lost in transfer), three subjects were excluded from the imaging analysis (n = 25), one subject was excluded from response time analysis (n = 27), and one subject's preference data were lost (n = 27).

Stimuli

The visual conditioned stimuli (CS) were complex abstract fractal images, and the unconditioned stimuli (UCS) were photographs of human male and female faces, attractive and unattractive. A set of 148 faces were previously rated by a separate group for attractiveness on a scale from 1 to 7 (O'Doherty et al. 2003b). Based on these ratings, eight faces were chosen to make up each of four conditions: female-attractive, male-attractive, female-unattractive, and male-unattractive. The faces had forward head position, and gazed forward with neutral to mildly happy expressions. The face images were masked to remove hair, were adjusted to be of approximately equal size and luminance, and centered in a 450 x 450-pixel gray background. We also used six abstract fractals, each centered in a170 x 170-pixel gray background. Stimuli were presented at a screen resolution of 800 x 600. Example stimuli are shown in the time course of a conditioning trial in Fig. 1A, and additional example face stimuli are shown in Fig. 1B. Stimuli were presented using Cogent 2000 developed by the Cogent 2000 team at the Functional Imaging Laboratory and the Institute of Cognitive Neuroscience and Cogent Graphics developed by John Romaya at the Laboratory of Neurobiology at the Wellcome Department of Imaging Neuroscience.

View larger version (34K): [in this window] [in a new window]   FIG. 1. A: sample time course of a reinforced conditioned stimuli (CS)+ conditioning trial with a female attractive face. B: example face stimuli.

SEXUAL ORIENTATION. Subjects first completed a questionnaire in which they were asked to describe their sexual orientation by choosing from a set of labels (heterosexual, homosexual, bisexual, transgender, polyamorous, none). They were also asked to rate on a 10-point scale how interested they are in having sex with men and women and how sexually attractive they find men and women.

BEHAVIORAL MEASURES OF LEARNING AND PREFERENCE MODULATION. During conditioning, attractive and unattractive faces were paired with affectively neutral fractal pictures. Subjects were first exposed to the fractal stimuli before conditioning to obtain pleasantness ratings and preference rankings. Pleasantness ratings were performed once subjects were installed in the scanner before fMRI data collection began. Subjects were first shown a screen with six fractal images; they were then shown each of the fractals once in random order and asked to verbally report a pleasantness rating between –10 and +10 (where –10 = very unpleasant, 0 = neutral, 10 = very pleasant). Next, subjects were presented with pairs of fractals and asked to indicate which of the two they preferred by pressing the left or right button on a two-button pad. Pairs were presented in random order with each combination presented three times and fractals randomly assigned to the left or right side of the screen. Subjects responded to a total of 45 pairs with each fractal appearing a total of 15 times.

During the conditioning procedure, subjects were asked to respond with a button press to indicate which side of the screen the fractal stimulus was presented on each trial. During conditioning, subjects were presented with each fractal a total of 48 times. These reaction times provided an additional on-line measure of conditioning (O'Doherty et al. 2006).

After conditioning, the preference and ratings tasks were repeated in that order; subjects were given the additional instruction that they should not try to match their previous answers but rather respond according to their present evaluation.

To assess explicit awareness of the contingencies, subjects were shown each of the six fractals in random order and asked how likely they thought it was that the fractal had been paired with an attractive face, using a scale from 0 to 10 (where 0 = not at all likely, and 10 = very likely). Subjects were also asked how unlikely it was that a stimulus was paired with an attractive face. We then asked subjects how likely and unlikely it was that each fractal had been paired with an unattractive face.

EVALUATION OF ATTRACTIVENESS OF FACE STIMULI. The final task in the experiment was to evaluate the attractiveness of the faces. Subjects were presented with each of the 32 faces in random order and asked to verbally report a subjective rating of facial attractiveness on a scale from –10 to +10.

BEHAVIORAL DATA ANALYSIS. We used differential ratings and preferences as an index of conditioning. We hypothesized that fractals paired with attractive faces would increase in pleasantness and become more preferred over fractals paired with unattractive faces especially for fractals paired with faces opposite in gender to the subject. A preliminary inspection of these data indicated that they were not normally distributed. Consequently, we used nonparametric statistics for all behavioral analyses in this study.

The pair-wise preference results were ranked based on the number of times a fractal was chosen as preferred, and category differences in ranking changes before and after conditioning were compared (e.g., change in rankings for fractals paired with attractive female faces compared with unattractive female faces).

Neuroimaging

CONDITIONING PROCEDURE. Four of the fractals were randomly assigned to be paired with faces from one of the four face gender/attractiveness categories, and two were assigned to never be paired with any faces. The fractal/face categories were: attractive female, attractive male, unattractive female, unattractive male, and unpaired. Each trial began with the presentation of a fractal image, randomly displayed either to the left or right of a central fixation cross. This fractal remained on the screen for 1.5 s. On reinforced CS+ trials, after 1 s, a picture of a face appeared in the middle of the screen next to the fractal. The two appeared together for 500 ms, the fractal then disappeared while the face remained on the screen for another full second, followed by a fixation cross for 500 ms. The duration of each trial was 3 s with the face and fractal each presented for 1.5 s with 0.5 s of overlap. We chose to use a delay conditioning paradigm with a short interstimulus interval to maximize conditioning efficacy. The time course of a CS+ trial is shown in Fig. 1. To enhance conditioning, the first three trials of each condition were reinforced CS+ trials in which the face followed the fractal, whereas for the remainder of the experiment, 50% of trials were reinforced. To obtain a trial-based behavioral measure of learning during the study and also to ensure that subjects attended to the task, subjects were instructed to press the left or right button on a two button pad, to indicate which side of the screen the fractal appeared. They were also instructed to keep their attention directed toward the center of the screen throughout the experiment. There were 48 trials of each type, 50% of which were reinforced, and each of the eight faces in a category was presented up to three times. Along with the 288 conditioning trials, we included a set of 96 null events, during which the fixation cross was presented for 3 s, to mimic the effect of a jittered intertrial interval and facilitate separation of neural responses from consecutive trials. Trials were randomly ordered, and the total duration of the conditioning session was 20 min.

PREDICTION ERROR SIGNALS. We used a simple trial-based Rescorla-Wagner rule to model trial-by-trial prediction errors in learning (Rescorla and Wagner 1972). This model uses a prediction error signal , which reflects the difference between the value of the outcome received on a given trial (R) and the value of the expected outcome on that trial (V): = R –V. The expected value V is then updated by adding weighted by a learning rate : V = V + .

In a follow-up region of interest analysis, we employed a real-time extension of the Rescorla-Wagner learning rule, a temporal difference model (Sutton 1988; Sutton and Barto 1990), in which the prediction error shifts backward in time from the face presentation to the cue presentation. We divided trials of each type in to early, middle, and late epochs and modeled the prediction error signal at the time of the face, 0.5 s before the face, and time of cue (1 s before the face), respectively.

The specific values used in our implementations of the model were the following: we modeled the presentation of a face with R = 1, the omission of a face with R = 0 (for both attractive and unattractive faces) and derived the learning rate () from subjects' behavioral responses. We used reaction times (responses to the conditioned stimuli) as a trial-by-trial measure of learning to derive model parameters from subjects' behavior. Reaction times have previously been shown to be modulated as a function of conditioning, and changes in reaction times over time have previously been found to correlate with reinforcement learning models (Critchley et al. 2002; Gottfried et al. 2002, 2003; Seymour et al. 2004). We derived learning signals for each subject based on their individual conditioning histories for a range of learning rates (ranging from 0.01 to 0.5). For each type of trial, we averaged log-adjusted trial-by-trial response times across subjects and fit these to a regression model that included the averaged learning signal curves. To account for general changes in reaction time that would occur over the experiment, we included an additional regressor as a covariate of no interest that reflected the change in reaction times across the experiment in the neutral trials (specifically a spline-smoothed fit of the averaged reaction times from the unpaired trials). This method allowed us to determine the learning rates that gave the best fit to subjects' behavior (on average across subjects). We used the learning rates resulting from this procedure to model the fMRI data for all subjects, by regressing these signals against the brain imaging data as described below (the specific learning rates are given in RESULTS).

FMRI SCANNING PROCEDURE. fMRI data were acquired on a Siemens AG (Erlangen, Germany) 3T TRIO MRI scanner; blood-oxygenation-level-dependent (BOLD) contrast was measured with gradient echo T2* weighted echo-planar images (EPI). Imaging parameters were optimized to minimize signal dropout in medial ventral prefrontal and anterior ventral striatum: we used a tilted acquisition sequence at 30° to the AC-PC line (Deichmann et al. 2003), and an eight-channel phased array coil that yields an 40% signal increase in this area over a standard coil. The first 5 volumes of 620 were discarded to permit T1 equilibration. Other parameters were as follows: in-plane resolution, 3 x 3 mm; slice thickness, 3 mm; repetition time, 2 s; echo time, 30 ms; field of view, 192 x 192 mm. A T1 weighted structural image was also acquired for each subject.

IMAGING DATA ANALYSIS. fMRI data were preprocessed in SPM2 (http://www.fil.ion.ucl.ac.uk/spm/software/spm2/). Images were corrected for slice acquisition time within each volume, motion corrected by aligning to the first volume (Friston et al. 1995) and unwarped to correct for estimated movement-related deformations in the EPI field (Andersson et al. 2001). They were normalized to a standard EPI template in Montreal Neurological Institute space, and spatial smoothing was applied using a Gaussian kernel with full width at half-maximum of 8 mm. The normalization parameters estimated for each subject were also applied to their T1-weighted structural scans.

Statistical analysis was carried out using the general linear model with the canonical hemodynamic response function (HRF) as a basis set. We describe results from two main analyses, designed to examine stimulus driven effects and learning related effects respectively. In the first analysis, fractal and face presentation events were modeled as delta functions. In the second analysis, prediction error signals were entered as parametric regressors for each trial type at the time a face would be presented, independently of whether a face had actually been shown. For all models, the six ongoing motion parameters estimated during realignment were included as regressors of no interest. The results from each subject were taken to the random effects level by applying t-tests between contrast images to produce group statistical parametric maps. We focused our analyses on brain regions of interest, specifically the striatum, orbitofrontal cortex and amygdala.

	RESULTS

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION GRANTS ACKNOWLEDGMENTS REFERENCES

 
Behavioral measures

FACE ATTRACTIVENESS RATINGS. Consistent with previous studies using the same set of faces (O'Doherty et al. 2003b), subjects rated the faces in the attractive category as significantly more attractive than those in the unattractive category, for both female (Wilcoxon signed-rank test |Z| = 4.264, n = 28, P < 0.001) and male faces (|Z| = 4.623, n = 28, P < 0.001). Gender differences were observed in evaluations of male faces as female subjects rated them as significantly more attractive than did male subjects (Mann-Whitney |Z| = 2.374, n = 15, 13, P < 0.05 and |Z| = 2.097, n = 15, 13, P < 0.05, attractive and unattractive, respectively). There were no significant gender differences in evaluations of female faces. Attractiveness evaluations are shown in Fig. 2A.

View larger version (23K): [in this window] [in a new window]   FIG. 2. Behavioral results. A: evaluations of attractiveness of faces in each category (attractive and unattractive, male and female), averaged across the 8 faces in each category. For both face genders, the unattractive mean was subtracted from the attractive mean and the differences averaged across subjects in 3 groups: all subjects, males, and females. |, SE. B: difference in pleasantness ratings for fractals pre- and postconditioning, unattractive difference subtracted from attractive difference, and this difference averaged across subjects in 3 groups: all subjects, males, and females. |, SE. , differences that are significantly (Wilcoxon signed ranks test P < 0.05). C: difference in number of times fractal was chosen as preferred pre- and postconditioning, unattractive difference subtracted from attractive difference, and this difference averaged across subjects in 3 groups: all subjects, males, and females. |, SE. , differences that are significant (Wilcoxon signed-rank test P < 0.05).

CHANGES IN PLEASANTNESS RATINGS OF STIMULI AS A FUNCTION OF CONDITIONING. Significant differences in pleasantness ratings for fractal stimuli were found from before to after conditioning for the stimuli paired with attractive female faces (Wilcoxon signed-rank test, |Z| = 2.169, n = 28, P < 0.05) across all subjects (both male and female). This effect was also significant in the subgroup of male subjects (Wilcoxon signed-rank test, |Z| = 1.992, n = 13, P < 0.05) but not female subjects. We did not find a similar effect for the fractals paired with male faces in any of the subject groups. Male and female subjects showed no significant differences in pleasantness ratings. Figure 2B shows differences in pleasantness ratings from before to after conditioning for stimuli paired with attractive and unattractive faces plotted for all subjects and males and females separately.

CHANGES IN BEHAVIORAL PREFERENCE FOR FRACTAL STIMULI AS A FUNCTION OF CONDITIONING. In male subjects, the increase in preferences for fractals paired with highly attractive female faces was significantly greater than for those paired with unattractive female faces (Wilcoxon signed-rank test, |Z| = 2.428, n = 13, P < 0.05), although this effect was not significant across all subjects. On the other hand, we did not find a similar effect in female subjects for those fractals paired with male faces. No significant effects were found in either male or female subjects for stimuli paired with same-sex attractive faces, and no significant gender differences in preference ratings were observed. Figure 2C shows differences in preference rankings for the fractal stimuli as a function of conditioning, plotted separately for stimuli paired with male and female faces, and groups of all, male and female subjects.

CORRELATIONS BETWEEN REACTION TIMES AND LEARNING MODEL. Our regression analysis showed that the Rescorla-Wagner model with the best-fitting learning rate was significantly correlated with changes in subjects' reaction time data over the experiment for all four trial types in which subjects learned the predictive value of the fractal cues. The learning rates obtained for each trial type were [attractive female: 0.026 (R² = 0.59, P < 0.05), attractive male: 0.04 (R² = 0.43, P < 0.05), unattractive female: 0.038 (R² = 0.48, P < 0.05), unattractive male: 0.04 (R² = 0.57, P < 0.05)]. Subject averaged reaction times are shown separately for low (0.0–0.2)- and high (0.2–0.5)-value trials for each condition in Fig. 3. This figure shows that for all four face-paired conditions, in both genders, there is a slowing in reaction times as model-predicted reward value increases. The mean reaction times for each condition are 469.19 ± 88.66 (SE) ms (attractive female) 462.42 ± 87.38 ms (attractive male) 465.16 ± 87.90 ms (unattractive female), and 460.33 ± 86.99 ms (unattractive male).

View larger version (23K): [in this window] [in a new window]   FIG. 3. Relationship between reaction times and predicted value from the Rescorla-Wagner learning model. Individual subjects' reaction times (RTs) were corrected for drift by taking the residuals from a regression onto the averaged reaction times for the neutral (never paired) conditions. Corrected RTs were then binned according to the predicted value derived from the Rescorla-Wagner learning model, using the derived learning rates for each trial type. The RTs were binned into low (0–0.2)- and high (0.2–0.5)-value trials. The plot shows that trials high in value show increased RTs compared with trials low in value.

PREDICTION ERROR CONTRASTS. To identify brain regions responding to prediction errors during learning with attractive compared with unattractive faces, we performed a linear contrast between prediction error signals for attractive and unattractive faces.

We found significant activation in one of our a priori regions of interest: nucleus accumbens (NAcc) as shown in Fig. 4 A ([–9 15 –3], z = 3.38 and [9 15 –9], z = 3.12; significant at P < 0.001, uncorrected). These areas survive small volume correction using a sphere of 8-mm radius defined around coordinates derived from a previous demonstration of reward prediction error activity in the NAcc ([–11 11 –2] and [11 11 –2]) (Knutson et al. 2005). The peak in the left NAcc is also significant in the contrast of learning with opposite-sex attractive compared with opposite-sex unattractive faces in all subjects ([–9 15 –6], z = 3.52; P < 0.001 uncorrected), and the subset of heterosexual subjects ([–9 15 –6], z = 3.79; P < 0.001 uncorrected). The contrast between learning with same-sex attractive compared with unattractive faces did not show any significant activations. Activations for prediction error contrasts are shown in Table 1.

View larger version (57K): [in this window] [in a new window]   FIG. 4. Prediction error related activity in the nucleus accumbens. A: voxels in the nucleus accumbens were significantly activated in a contrast of prediction error signals for attractive faces vs. unattractive faces, voxels in yellow are significant at P < 0.001, voxels in red are significant at P < 0.01. B: parameter estimates for prediction error at the peak nucleus accumbens voxel from the attractive-unattractive contrast [–9 15 –3], averaged across subjects in 3 groups: all subjects, males, and females. |, SE. , differences that are significant (1-tailed t-test, P < 0.05). C: subject averaged time courses, aligned to the beginning of a trial, i.e., onset of the fractal cue; faces were presented at 1 s. |, SE. Time courses extracted from each subject's peak voxel in the left NAcc region. Left: averaged over attractive and unattractive trials, unpaired trials subtracted from paired. Middle and right: paired and unpaired trials separately for attractive and unattractive faces, respectively.

In the contrast of prediction error for learning with attractive compared with unattractive faces, we also found significant effects in the right and left inferior frontal gyrus (see Table 1). These areas remain significant when we restrict this contrast to opposite-sex faces.

TEST FOR LEARNING RELATED CHANGES OVER TIME. To establish whether activity in NAcc is associated with a temporally evolving learning signal as opposed to nonlearning-related effects induced by the presence or absence of a face, we performed an additional analysis on the time-series data extracted from the peak voxel in NAcc (at [9 15 –9]). For this, we included in the analysis a prediction error regressor that temporally shifted from the time of face presentation to the time of cue presentation, using a real-time extension of the Rescorla Wagner learning rule: temporal difference learning (Sutton and Barto 1990). We included in the same analysis a regressor at the time of face presentation, only when faces were actually presented. The temporal difference prediction error signal was a significantly better fit to activity in the NAcc than the face regressor at P < 0.05, suggesting that activity in this structure reflects dynamic learning related changes and not merely effects relating to the presence of absence of a face.

SEPARATE PREDICTION ERRORS DURING LEARNING WITH ATTRACTIVE AND UNATTRACTION FACES. Although comparing prediction error responses during learning with attractive and unattractive faces produced robust differences, due to the opposing direction of the prediction error signal, we also examined learning signals in response to both attractive and unattractive faces independently. A simple contrast for areas showing a positive correlation with the prediction error signal for attractive faces produced a peak in the right NAcc (at [6 15 –12]) that survived correction for small volume at P < 0.05 FDR corrected in a 5-mm sphere centered on the peak identified above ([9 15 –9]). A simple contrast to detect areas showing a negative correlation with the prediction error signal for unattractive faces, also produced a peak in the left NAcc (at [–6 12 –3]) which survived correction for small volume at P < 0.05 FDR corrected in a 5 -mm sphere centered on the NAcc peak identified above ([–9 15 –3]).

We found evidence for a positive correlation with prediction error signals during learning with both attractive and unattractive faces in the amygdala, another of our a-priori regions of interest: for attractive faces in right amygdala ([24 0 –25], z = 3.31 P < 0.001 uncorrected) and for unattractive faces in right amygdala ([18 –6 –21], z = 4.02 P < 0.001 uncorrected) and left amygdala ([-18 –6 –18], z = 3.26 P < 0.001 uncorrected).

TIME-COURSE PLOTS. We also extracted time courses for peak voxels in the region of individual subjects' left NAcc, and averaged over attractive and unattractive trials, subtracting the averaged trials for which no face was presented from the averaged trials for which a face was presented. The resulting time courses are shown in Fig. 4C (left) and indicate a positive increase in BOLD signal for positive prediction errors and decrease in signal for negative prediction errors. Figure 4C (middle) shows that the NAcc responds positively to the presentation of an attractive face and negatively to the omission of an attractive face, whereas Fig. 4C (right) shows that for unattractive faces this relationship is inverted with increased activation seen to the omission of a face.

PREDICTION ERROR RESPONSES TO LEARNING WITH SAME AND OPPOSITE-SEX FACES. We explored prediction error responses to fractals paired with opposite- and same-sex faces by conducting a post hoc statistical analysis on the contrast estimates derived from the left ventral striatum (plotted in Fig. 4B) in heterosexual subjects. In male subjects, we found a significant difference in responses for attractive compared with unattractive female faces (|t| = 3.01, dof = 8, P < 0.05), but no difference for male faces, whereas in female subjects, we found a significant difference in contrast estimates for attractive compared with unattractive male faces (|t| = 3.16, dof = 12, P < 0.05) but not female faces. Pooling male and female subjects we found a significant effect of attractiveness when subjects were presented with opposite (|t| = 4.31, dof = 21, P < 0.001) but not same-sex faces.

MAIN EFFECT OF ATTRACTIVENESS. We also tested for regions responding to receipt of the attractive faces themselves. For this, we performed a linear contrast of attractive-unattractive faces at the random effects level (Fig. 5 A) and found significant effects in medial OFC ([12 39 –9] z = 2.93) extending into medial prefrontal cortex, a region previously shown to be responsive to the receipt of attractive faces (Aharon et al. 2001;O'Doherty et al. 2003b). The OFC area survived correction for small volume at P < 0.05 FDR corrected in an 8-mm sphere centered around coordinates from a previous study of facial attractiveness (at [16 45 –11]) (from Aharon et al. 2001).

View larger version (34K): [in this window] [in a new window]   FIG. 5. Main effect of attractiveness in orbitofrontal cortex. A: voxels in the orbitofrontal cortex extending into medial prefrontal cortex were significantly activated in a contrast of attractive faces vs. unattractive faces. The peak in medial OFC ([12 39 –9] z = 2.93), survived correction for small volume at P < 0.05 FDR corrected in an 8-mm sphere centered around coordinates from a previous study (see RESULTS). For visualization, the threshold is set at P < 0.01 uncorrected. B: subject averaged time courses, aligned to the beginning of a trial, i.e., onset of the fractal cue; faces were presented at 1 s. |, SE. Time courses extracted from the medial OFC peak in response to the main effect of attractiveness. Left: averaged over attractive and unattractive trials, unpaired trials subtracted from paired. Middle and right: paired and unpaired trials separately for attractive and unattractive faces, respectively.

	DISCUSSION

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION GRANTS ACKNOWLEDGMENTS REFERENCES

By measuring neural activity with fMRI while subjects acquired this association, we were able to observe learning-related activity in the brain as the association was formed. We found that reward-prediction errors were engaged in the ventral striatum, differentially for stimuli paired with attractive compared with unattractive faces. Prediction errors have been observed during learning with other types of reward, such as juice and money (Delgado et al. 2000; Tobler et al. 2006). The observation that attractive faces also engage these signals further reinforces the proposal that attractive faces can be considered to be a form of visual reward (Aharon et al. 2001; O'Doherty et al. 2003b). The present result also provides insight into the mechanism by which attractive faces transfer their rewarding properties to other stimuli.

It is notable that increases in activity occurred in the striatum in response to positive prediction error signals after the unexpected presentation of an attractive face, but the opposite effect was found in response to the unexpected presentation of an unattractive face in which case a decrease in signal was observed. These findings suggest that ventral striatum shows a very different response profile to prediction error signals during learning with attractive as opposed to unattractive faces. These results are similar to effects found for prediction error signals generated during learning with monetary reward and punishment (Kim et al. 2006). These results are also compatible with response profiles reported in striatum in fMRI studies involving delivery of monetary reward and punishment, whereby ventral striatum has been shown to increase in response to receipt of monetary reward and decrease in response to receipt of monetary loss (Delgado et al. 2000). By contrast, a different pattern of responses has been observed in ventral striatum in response to prediction errors produced during learning with other types of reinforcers such as somatosensory pain and even nonpreferred flavors (O'Doherty et al. 2006; Seymour et al. 2004). In these cases, an increase in signal has been reported in ventral striatum after the unexpected delivery of a cue signaling subsequent pain or unpleasant flavor. Thus ventral striatum appears to show very different neural responses as a function of learning with different types of reinforcers. This raises the question as to the nature of the difference between reinforcers that leads to such divergent response profiles. One possibility is that ventral striatum responds differently to learning with primary as opposed to secondary reinforcers. Money can be considered to be a secondary or learned reinforcer, whereas pain and food can be argued to be primary reinforcers (Rolls 2000). However, facial attractiveness is often suggested to be a primary reinforcer as judgments of facial attractiveness are suggested to be culturally invariant (Cunningham et al. 1995), and attractiveness has been argued to signal reproductive fitness (Rhodes et al. 2005). As a consequence, the fact that attractive faces and money are similar in the way they activate the striatum would appear to argue against a primary versus secondary reinforcer account of differential striatal function. An alternative possibility is that ventral striatum is involved not in learning about the sensory properties or abstract value of unconditioned stimuli but instead learns associations between arbitrary stimuli and the unconditioned responses produced by an unconditioned stimulus. Differences in the nature of the unconditioned responses produced by different reinforcers could potentially account for differential activity in the striatum. Future studies will be needed to investigate this possibility further.

Although we found an overall effect of attractiveness on prediction error activity in the ventral striatum, we also found that in this area the effect was significant when heterosexual subjects were presented with opposite-sex faces but not same-sex faces. That is, prediction error responses were enhanced when learning about attractive faces of the opposite sex in both genders. This suggests that ventral striatum may be involved in mediating learning about attributes linked to sexual preference as opposed to learning about more general aspects of visual esthetics (Kranz and Ishai 2006).

In contrast to the nucleus accumbens, the amygdala showed positive correlations with prediction error signals during learning with both attractive and unattractive faces, consistent with previous findings of a role for amygdala in conditioning involving both appetitive and aversive stimuli (Buchel 1998; Gottfried et al. 2003; LaBar 1998; Paton et al. 2006). More generally, these results add to an extensive prior literature implicating the amygdala in processing stimuli of both positive and negative valence (Garavan et al. 2001; Hamann et al. 1999; Winston et al. 2003).

The results of this study also have important implications for understanding the underlying mechanisms by which product advertising can influence behavioral preference in the marketplace. Marketers have long attempted to bias consumer preference by pairing a particular product with another stimulus that is already highly valued, such as an attractive face. Indeed, changes in product evaluations and preference have been observed in behavioral experiments as a function of such pairing procedures (Gorn et al. 1993; Smith and Engel 1968). However, the precise mechanism by which preference modulation takes place has remained an open question. One possibility is that changes in preference evaluations occur through cognitive appraisal or top down modulation of affect (as in cognitive appraisal) (cf. Folkman et al. 1986). Another possibility is that preference evaluations occur as a function of classical conditioning (Baeyens et al. 1993). We directly tested this hypothesis using a classical conditioning study that, compared with previous neuroimaging studies of classical conditioning (Gottfried et al. 2002; Tobler et al. 2006), used a relatively long (1.5 s) UCS duration. Although studies examining the effect of UCS duration have shown mixed results (Bitterman et al. 1952; Kawai and Imada 1996; Tait et al. 1983), we chose a longer UCS presentation to increase the salience of the face stimuli. Our results provide evidence that the change in preference likely occurs as a function of classical conditioning by showing that similar neural mechanisms are engaged during evaluative preference modulation as are engaged during other types of classical conditioning. Moreover, the fact that evaluative preference modulation specifically engages prediction error signals in the ventral striatum, suggests that this procedure may recruit dopamine neurons in the midbrain as is known to be the case during learning with other kinds of reward in non-human primates (Hollerman and Schultz 1998). Consistent with the preceding suggestion, a recent fMRI study has shown that prediction error signals expressed in the ventral striatum during reward-learning can be modulated through pharmacological manipulation of dopamine levels in humans (Pessiglione et al. 2006), indicating that the source of such signals in human fMRI studies may in part be attributable to the afferent input from dopamine neurons.

To conclude, in the present study, we have found significant prediction error-related activity in the ventral striatum during conditioning with attractive compared unattractive faces. These findings suggest that an arbitrary stimulus can acquire conditioned value by being paired with pleasant visual stimuli just as with other types of reward like money or juice. Such a learning process elicits a reward-prediction error signal in a main target structure of dopamine neurons: the ventral striatum. The learning process we describe here may provide insights into the neural mechanisms used in advertising to influence behavioral preferences, whereby consumers are exposed repeatedly to simple associations between products and rewarding visual stimuli such as pretty faces.

	GRANTS

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION GRANTS ACKNOWLEDGMENTS REFERENCES

 
This research is supported by a grant to J. O'Doherty from the Gimbel Discovery Fund in Neuroscience.

	ACKNOWLEDGMENTS

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION GRANTS ACKNOWLEDGMENTS REFERENCES

 
We thank M. Tyzska and S. Flaherty for assistance with the functional imaging.

	FOOTNOTES

 
The costs of publication of this article were defrayed in part by the payment of page charges. The article must therefore be hereby marked "advertisement" in accordance with 18 U.S.C. Section 1734 solely to indicate this fact.

Address for reprint requests and other correspondence: J. O'Doherty, Computation and Neural Systems, California Institute of Technology, 1200 E. California Blvd., Pasadena, CA 91125 (E-mail: jdoherty{at}hss.caltech.edu)

	REFERENCES

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION GRANTS ACKNOWLEDGMENTS REFERENCES

 
Adolphs R. The neurobiology of social cognition. Curr Opin Neurobiol 11: 231–239, 2001.[CrossRef][ISI][Medline]

Aharon I, Etcoff N, Ariely D, Chabris CF, O'Connor E, Breiter HC. Beautiful faces have variable reward value: fMRI and behavioral evidence. Neuron 32: 537–551, 2001.[CrossRef][ISI][Medline]

Andersson JL, Hutton C, Ashburner J, Turner R, Friston K. Modeling geometric deformations in EPI time series. Neuroimage 13: 903–919, 2001.[ISI][Medline]

Baeyens F, Hermans D, Eelen P. The role of CS-US contingency in human evaluative conditioning. Behav Res Ther 31: 731–737, 1993.[CrossRef][ISI][Medline]

Baker MJ, Churchill GAJ. The impact of physically attractive models on advertising evaluations. J Market Res XIV: 538–555, 1977.

Bitterman ME, Reed P, Krauskopf J. The effect of the duration of the unconditioned stimulus upon conditioning and extinction. Am J Psychol 65: 256–262, 1952.[CrossRef][ISI][Medline]

Blair RJR. Facial expressions, their communicatory functions and neuro-cognitive substrates. Philos Trans R Soc Lond B Biol Sci 358: 561–572, 2003.[CrossRef][ISI][Medline]

Buchel C. Brain systems mediating aversive conditioning: an event-related fMRI study. Neuron 20: 947–957, 1998.[CrossRef][ISI][Medline]

Cox SM, Andrade A, Johnsrude IS. Learning to like: a role for human orbitofrontal cortex in conditioned reward. J Neurosci 25: 2733–2740, 2005.[Abstract/Free Full Text]

Critchley HD, Mathias CJ, Dolan RJ. Fear conditioning in humans: the influence of awareness and autonomic arousal on functional neuroanatomy. Neuron 33: 653–663, 2002.[CrossRef][ISI][Medline]

Cunningham MR, Roberts AR, Barbee AP, Druen PB, Wu C-H. Their ideas of beauty are, on the whole, the same as ours: consistency and variability on the cross-cultural perception of female physical attractiveness. J Personality Social Psychol 68: 261–279, 1995.[CrossRef][ISI]

Deichmann R, Gottfried JA, Hutton C, Turner R. Optimized EPI for fMRI studies of the orbitofrontal cortex. Neuroimage 19: 430–441, 2003.[CrossRef][ISI][Medline]

Delgado MR, Nystrom LE, Fissell C, Noll DC, Fiez JA. Tracking the hemodynamic responses to reward and punishment in the striatum. J Neurophysiol 84: 3072–3077, 2000.[Abstract/Free Full Text]

Dion K, Bersheid E, Walster E. What is beautiful is good. J Personality Social Psychol 24: 285–290, 1972.[CrossRef][ISI][Medline]

Dipboye RL, Arvey RD, Terpstra DE. Sex and physical attractiveness of raters and applicants as determinants of resume evaluations. J Appl Psychol 4: 288–294, 1977.

Folkman S, Lazarus RS, Dunkel-Schetter C, DeLongis A, Gruen RJ. Dynamics of a stressful encounter: cognitive appraisal, coping, and encounter outcomes. J Personality Social Psychol 50: 992–1003, 1986.[CrossRef][ISI][Medline]

Friston KJ, Ashburner J, Poline JB, Frith CD, Heather JD, Frackowiak RS. Spatial registration and normalisation of images. Hum Brain Mapp 2: 165–189, 1995.

Garavan H, Pendergrass JC, Ross TJ, Stein EA, Risinger RC. Amygdala response to both positively and negatively valenced stimuli. Neuroreport 12: 2779–2783, 2001.[CrossRef][ISI][Medline]

Gorn GJ, Goldberg ME, Basu K. Mood, awareness, and product evaluation. J Consum Psychol 2: 237–256, 1993.[CrossRef]

Gottfried JA, O'Doherty JP, Dolan RJ. Appetitive and aversive olfactory learning in humans studied using event-related functional magnetic resonance imaging. J Neurosci 22: 10829–10837, 2002.[Abstract/Free Full Text]

Gottfried JA, O'Doherty J, Dolan RJ. Encoding predictive reward value in human amygdala and orbitofrontal cortex. Science 301: 1104–1107, 2003.[Abstract/Free Full Text]

Hamann SB, Ely TD, Grafton ST, Kilts CD. Amygdala activity related to enhanced memory for pleasant and aversive stimuli. Nat Neurosci 2: 289–293, 1999.[CrossRef][ISI][Medline]

Hamermesh DS, Biddle JE. Beauty and the labor market. Am Econom Rev 84: 1174–1194, 1994.

Haxby JV, Hoffman EA, Gobbini IM. The distributed human neural system for face perception. Trends Cogn Sci 4: 223–233, 2000.[CrossRef][ISI][Medline]

Hollerman JR, Schultz W. Dopamine neurons report an error in the temporal prediction of reward during learning. Nat Neurosci 1: 304–309, 1998.[CrossRef][ISI][Medline]

Ishai A, Schmidt CF, Boseiger P. Face perception is mediated by a distributed cortical network. Brain Res Bull 67: 87–93, 2005.[CrossRef][ISI][Medline]

Johnsrude IS, Owen AM, White NM, Zhao WV, Bohbot V. Impaired preference conditioning after anterior temporal lobe resection in humans. J Neurosci 20: 2649–2656, 2000.[Abstract/Free Full Text]

Kanwisher N, McDermott J, Chun M. The fusiform face area: a module in human extrastriate cortex specialized for the perception of faces. J Neurosci 17: 4302–4311, 1997.[Abstract/Free Full Text]

Kawai N, Imada H. Between- and within-subject effects of US duration on conditioned suppression in rats: contrast makes otherwise unnoticed duration dimension stand out. Learn Motiv 27: 92–111, 1996.[CrossRef][ISI]

Kim H, Shimojo S, O'Doherty JP. Is avoiding an aversive outcome rewarding? Neural substrates of avoidance learning in the human brain. PLoS Biol 4: 1453–1461, 2006.[ISI]

Knutson B, Taylor J, Kaufman M, Peterson R, Glover G. Distributed neural representation of expected value. J Neurosci 25: 4806–4812, 2005.[Abstract/Free Full Text]

Kranz F, Ishai A. Face perception is modulated by sexual preference. Curr Biol 16: 63–68, 2006.[CrossRef][ISI][Medline]

LaBar KDM. Human amygdala activation during conditioned fear acquisition and extinction: a mixed-trial fMRI study. Neuron 20: 937–945, 1998.[CrossRef][ISI][Medline]

McClure SM, Berns GS, Montague PR. Temporal prediction errors in a passive learning task activate human striatum. Neuron 38: 339–346, 2003.[CrossRef][ISI][Medline]

O'Doherty JP, Buchanan TW, Seymour B, Dolan RJ. Predictive neural coding of reward preference involves dissociable responses in human ventral midbrain and ventral striatum. Neuron 49: 157–166, 2006.[CrossRef][ISI][Medline]

O'Doherty JP, Dayan P, Friston K, Critchley H, Dolan RJ. Temporal difference models and reward-related learning in the human brain. Neuron 38: 329–337, 2003a.[CrossRef][ISI][Medline]

O'Doherty J, Dayan P, Schultz J, Deichmann R, Friston K, Dolan RJ. Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304: 452–454, 2004.[Abstract/Free Full Text]

O'Doherty J, Winston J, Critchley H, Perrett D, Burt DM, Dolan RJ. Beauty in a smile: the role of medial orbitofrontal cortex in facial attractiveness. Neuropsychologia 41: 147–155, 2003b.[CrossRef][ISI][Medline]

Olson IR, Marshuetz C. Facial attractiveness is appraised at a glance. Emotion 5: 186–201, 2005.

Paton JJ, Belova MA, Morrison SE, Salzman CD. The primate amygdala represents the positive and negative value of visual stimuli during learning. Nature 439: 865–870, 2006.[CrossRef][Medline]

Pessiglione M, Seymour B, Flandin G, Dolan RJ, Frith CD. Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans. Nature 442: 1042–1045, 2006.[CrossRef][Medline]

Rescorla RA, Wagner AR. A theory of Pavlovian conditioning: Variations in effectiveness of reinforcement and nonreinforcement. In: Classical Conditioning II: Current Research and Theory, edited by Black AH, Prokasy WF. New York: Appleton Century-Crofts, 1972.

Rhodes G. The evolutionary psychology of facial beauty. Annu Rev Psychol 57: 199–226, 2006.[CrossRef][ISI][Medline]

Rhodes G, Simmons LW, Peters M. Attractiveness and sexual behavior: does attractiveness enhance mating success? Evol Hum Behav 26: 186–201, 2005.[CrossRef][ISI]

Rolls ET. The orbitofrontal cortex and reward. Cereb Cortex 10: 284–294, 2000.[Abstract/Free Full Text]

Schultz W, Dayan P, Montague PR. A neural substrate of prediction and reward. Science 275: 1593–1599, 1997.[Abstract/Free Full Text]

Seymour B, O'Doherty JP, Dayan P, Koltzenburg M, Jones AK, Dolan RJ, Friston KJ, Frackowiak RS. Temporal difference model describes higher-order learning in humans. Nature 429: 664–667, 2004.[CrossRef][Medline]

Smith GH, Engel R. Influence of a female model on perceived characteristics of an automobile. 76th Annual Convention American Psychological Association, San Francisco, CA, 1968.

Sutton RS. Learning to predict by the methods of temporal differences. Machine Learn 3: 9–44, 1988.

Sutton RS, Barto AG. Time derivative models of Pavlovian reinforcement. In: Learning and Computational Neuroscience: Foundations of adaptive networks, edited by Gabriel M, Moore J. Cambridge, MA: MIT Press, 1990, p. 497–537.

Tait RW, Kehoe EJ, Gormezano I. Effects of unconditioned stimulus-duration on classical-conditioning of the rabbits nictitating-membrane response. J Exp Psychol Anim Behav Process 9: 91–101, 1983.[CrossRef][ISI]

Thornhill H, Gangestad SW. Facial attractiveness. Trends Cogn Sci 3: 452–460, 1999.[CrossRef][ISI][Medline]

Tobler PN, O'Doherty JP, Dolan RJ, Schultz W. Human neural learning depends on reward prediction errors in the blocking paradigm. J Neurophysiol 95: 301–310, 2006.[Abstract/Free Full Text]

Whalen PJ, Rauch SL, Etcoff NL, McInerney SC, Lee MB, Jenike MA. Masked presentations of emotional facial expressions modulate amygdala activity without explicit knowledge. J Neurosci 18: 411–418, 1998.[Abstract/Free Full Text]

Winston JS, O'Doherty J, Dolan RJ. Common and distinct neural responses during direct and incidental processing of multiple facial emotions. Neuroimage 20: 84–97, 2003.[CrossRef][ISI][Medline]

This article has been cited by other articles:

				T. A. Hare, J. O'Doherty, C. F. Camerer, W. Schultz, and A. Rangel Dissociating the Role of the Orbitofrontal Cortex and the Striatum in the Computation of Goal Values and Prediction Errors J. Neurosci., May 28, 2008; 28(22): 5623 - 5630. [Abstract] [Full Text] [PDF]

				J. D. Beaver, A. D. Lawrence, L. Passamonti, and A. J. Calder Appetitive Motivation Predicts the Neural Response to Facial Signals of Aggression J. Neurosci., March 12, 2008; 28(11): 2719 - 2725. [Abstract] [Full Text] [PDF]

This Article