## Abstract

Evaluation of confidence about one's knowledge is key to the brain's ability to monitor cognition. To investigate the neural mechanism of confidence assessment, we examined a biologically realistic spiking network model and found that it reproduced salient behavioral observations and single-neuron activity data from a monkey experiment designed to study confidence about a decision under uncertainty. Interestingly, the model predicts that changes of mind can occur in a mnemonic delay when confidence is low; the probability of changes of mind increases (decreases) with task difficulty in correct (error) trials. Furthermore, a so-called “hard-easy effect” observed in humans naturally emerges, i.e., behavior shows underconfidence (underestimation of correct rate) for easy or moderately difficult tasks and overconfidence (overestimation of correct rate) for very difficult tasks. Importantly, in the model, confidence is computed using a simple neural signal in individual trials, without explicit representation of probability functions. Therefore, even a concept of metacognition can be explained by sampling a stochastic neural activity pattern.

- decision confidence
- lateral intraparietal cortex
- line-attractor neural model

a key to monitoring of cognition (metacognition) is our ability to evaluate the degree of confidence that we have about a decision, a strategy to tackle the problem at hand, a newly acquired piece of knowledge, etc. Confidence estimation has been an important subject of research in cognitive and developmental psychology (Flavell 1979; Vickers 1979). In laboratory studies, confidence can be measured using postdecision wagering (PDW), where subjects first perform a perceptual decision and then make a high-low bet between a risky option (associated with a high reward if the first-order choice is correct, a loss otherwise) and a safe option (associated with a low reward regardless of the first-order choice). Consequently, if subjects have less confidence about their choice, they would be more likely to bet on the low but certain reward option (Dienes and Seth 2010; Fleming and Dolan 2010; Fleming et al. 2010; Kepecs and Mainen 2012; Persaud et al. 2007). Recently, researchers have begun to use PDW and other task designs with behaving animals to explore the neural basis of confidence estimation (Kepecs et al. 2008; Lak et al. 2014; Middlebrooks and Sommer 2011, 2012; Smith et al. 2003).

In a monkey experiment, Kiani and Shadlen (2009) extended a well-known discrimination task to examine neural signals correlated with confidence. In this task, a subject is required to decide between two possible directions (indicated by two directional targets) of a random-dots motion stimulus. Specifically, Kiani and Shadlen used a fixed-duration (FD) version of the task (Shadlen and Newsome 2001), where the visual motion stimulus is followed by a delay period, and monkeys must indicate the decision at the end of the delay by a saccadic response to one of the directional targets. In a random subset of trials, they offered a third target (*T*_{s}) during the delay period, and monkeys could opt to *T*_{s} for a certain but small amount of reward. Interestingly, monkeys selected *T*_{s} more often when motion strength was weaker or stimulus duration became shorter, under which conditions the error rate was higher and selecting *T*_{s} gave rise to an improvement of performance across trials. The probability of choosing *T*_{s} (*P*_{sure}) thus reflected a degree of choice uncertainty. Importantly, *P*_{sure} was found to modulate single-neuron activity in lateral intraparietal (LIP) area that was correlated with accumulating decision evidence of a choice. This finding supports the intuitive idea that confidence signal is an integral part of a decision process (Vickers 1979) and reflected in a neural decision variable (Gold and Shadlen 2007).

Computational schemes have been proposed for the study of confidence (Kepecs and Mainen 2012; Kepecs et al. 2008; Kiani and Shadlen 2009; Moreno-Bote 2010; Pleskac and Busemeyer 2010; Ratcliff and Starns 2009; Rolls et al. 2010a, 2010b; Vickers 1979). In particular, with the use of drift diffusion model (DDM) (Ratcliff and Smith 2004), confidence has been defined in terms of the log posterior ratio for the two choices given the decision variable at the time of behavioral response (Kiani and Shadlen 2009). This DDM, nevertheless, has some limitation to account for the complexity of confidence (Van Zandt 2000), e.g., a decision variable that terminates at a fixed threshold may not present a graded confidence across trial (Kiani et al. 2014).

In this work, to uncover neural circuit mechanisms underlying confidence estimation, we took a different approach and employed a biophysically realistic cortical network model of spiking neurons, which was previously shown to successfully simulate the random-dots motion-direction discrimination experiment (Furman and Wang 2008). We investigated whether the same model could accurately reproduce the salient findings from Kiani and Shadlen (2009). The model is endowed with a continuous network of neurons that can represent any direction; therefore, it can be readily extended to incorporate the presentation of a sure target (*T*_{s}) during a delay period. Notably, such a model of decision-making and memory processes was not originally designed for the Kiani-Shadlen experiment to account for confidence estimation. It is thus surprising that the model can capture both a range of behavioral performance data and physiological observations from LIP single neurons. We noted that neurons selective for *T*_{s} win the competition (thus *T*_{s} was chosen) when the differential activity of neurons selective for the two alternative choices is small. Quantitatively, we found that confidence could be estimated, at any time, as a sigmoid function of the differential firing activity of the two competing neural pools selective for the alternative choices. Therefore, choice confidence is computed simultaneously when a decision is made, and a trial-by-trial variation of choice is generated by sampling of stochastic neural dynamics (Wang 2008).

## MATERIALS AND METHODS

#### Network model.

We employed a spiking neural network model, which has been previously used to simulate a categorical decision of an analog feature, like motion direction (Furman and Wang 2008; Liu and Wang 2008). This model consists of 2,048 pyramidal cells and 512 interneurons. Both pyramidal cells and interneurons are modeled as integrate-and-firing neurons; excitatory postsynaptic currents from pyramidal cells are mediated by α-amino-3-hydroxy-5-methyl-4-isoxazolepropionic acid (AMPA) and *N*-methyl-d-aspartate (NMDA) receptors, whereas inhibitory postsynaptic currents from interneurons are mediated by GABA receptors. Pyramidal cells are uniformly placed on a ring according to their preferred motion directions and continuously span 360° of possible motion directions (Fig. 1*B*), while the interneurons constitute a nonselective neural pool. The recurrent connectivity strength between two pyramidal cells is a Gaussian function of the difference between their preferred motion directions, whereas those from and onto the interneurons are broad and uniform (Fig. 1*B*). All the cells receive a background noise mediated by AMPA receptors, which is modeled as uncorrelated Poisson spike train. We used the neuronal and synaptic parameters from Furman and Wang (2008), which are fully specified therein, with a change of the background noise rate to 2,200 Hz, which ensures a choice is generated even if the motion strength is weak and stimulus duration is short. With these parameters, the network is endowed with winner-take-all competition so that only one of the neural pools wins (reaching an average population firing rate >50 Hz for at least 50 ms), and the decision is maintained in the form of a bell-shaped persistent activity pattern (“bump attractor”) in a delay period.

#### Simulation protocol of FD discrimination-decision task.

Our model assumes that neurons in area LIP incorporate sensory evidence (Cook and Maunsell 2002; Hanks et al. 2006; Roitman and Shadlen 2002) and reward signals (Platt and Glimcher 1999; Sugrue et al. 2004; Tobler et al. 2005). For simplicity, we assumed that the amount of reward for each target (i.e., two directional targets and a sure target) is associated with the instantaneous input strength of its current at the moment of the target onset (Soltani and Wang 2006). That is to say, the amplitude of the sure target input does not correspond to its physical properties (like the luminance) in the experiment; instead it is related to the behavioral significance of the sure target that a monkey learned, i.e., the amount of reward it received by choosing the sure target (Fig. 8).

In an FD version of the two-alternative direction-discrimination task (Fig. 1*A*), two directional targets, *T*_{A} (90°) and *T*_{B} (270°), are first presented to the network. A random-dot stimulus with net motion to *T*_{A} is presented at 500 ms after target onset. The difficulty of the task is modulated by the stimulus duration (randomly chosen from 110, 130, 152, 178, 208, 244, 289, 348, 439, and 627 ms) and the percentage of coherently moving dots (the motion strength). We modeled the external input to pyramidal cell *i* (at θ_{i}) as a sum of two target signals, *I*^{tar}_{i}(*t*) (*tar* = {*A*, *B*}; Fig. 1*D*, black line), and the motion stimulus, *I*^{m}_{i}(*t*) (Fig. 1*C*). The target inputs to *T*_{A} and *T*_{B} are identical:

*I*^{tar}_{i}(*t*) = *I*^{tar}(*t*)exp[−(θ_{i} − θ_{tar})^{2}/2σ^{2}_{tar}]

where θ_{A} = 90°; θ_{B} = 270°; σ_{tar} = 10°.

*I*^{tar}(*t*) = *I*_{1} + *I*_{2}exp[−(*t* − *t*_{d} − 200)/τ_{d}], *t*_{d} + 100 < *t* ≤ *t*_{m} + 80

*I*^{tar}(*t*) = *I*_{3} + (*I*_{1} − *I*_{3})exp[−(*t* − *t*_{m} − 80)/τ_{ss}], *t*_{m} + 80 < *t*

where *t*_{d} = 400 ms and *t*_{m} = 800 ms are the onset times of targets and motion, respectively; τ_{d} = 500 ms and τ_{ss} = 15 ms are the time constants of the adaption and suppression, respectively; *I*_{1} = 250 pA, *I*_{2} = 50 pA, and *I*_{3} = 60 pA. Specially, the target (motion, respectively) input onset time is 100 (200, respectively) ms after the target (motion, respectively) onset time, and the target input is suppressed by the motion stimulus with a latency of 80 ms (Roitman and Shadlen 2002); with the high intensities of the target inputs, winner-take-all competition between the two targets does not take place before the motion stimulus onset (Furman and Wang 2008; Liu and Wang 2008; Wong et al. 2007).

In simulation, motion input is modeled to imitate the neural response in the middle-temporal area (MT) to the random-dot stimuli. We constructed such a population activity as a Gaussian function with a tuning width independent of motion strength while motion presented (*t*_{m} + 200 < *t* < *t*_{mo}, *t*_{mo} is the moment of motion input offset)

*I*^{m}(*i*) = *m*_{0} + *coh* {−*m*_{1} + *m*_{2}exp[−(θ_{i} − θ_{m})^{2}/2σ^{2}_{m}]}

where the motion strength 0 ≤ *coh* ≤ 1; net direction θ_{m} = 90°; σ_{m} = 40°. We kept the activity normalized, i.e., <*I*^{m}(*i*)> = *m*_{0} = 4 pA; *m*_{1} = 4.93 pA; *m*_{2} = 25 pA.

In trials with *T*_{s} (θ_{s} = 180°), where there was the opt-out safe target presented (Fig. 1*D*, red line), we modeled its time-dependent current, *I*^{s}(*i*), as:

*I*^{s}(*i*) = *I*^{s}(*t*)exp[−(θ_{i} − θ_{s})^{2}/2σ^{2}_{tar}]

which is added to the external input. We used:

*I*^{s}(*t*) = *I*_{3} + *I*_{4}exp[−(*t* − *t*_{s} − *t*_{mo})/τ_{s}]

for *t* > *t*_{s} + *t*_{mo}, where *t*_{s} is *T*_{s} input onset time after the motion input offset, *t*_{mo}, with a latency of 100 ms to the network after *T*_{s} onset. In simulations, we used *t*_{s} = 575 ms; τ_{s} = 90 ms; *I*_{4} = 240 pA (see Fig. 8 for a discussion on the choice of *I*_{4}), expect Fig. 5 (*t*_{s} is equal to 575, 750, or 925 ms).

The network model is taken from Furman and Wang (2008), with a few parameter changes, i.e., background noise that ensures a choice is generated even if the motion strength is weak and stimulus duration is short, and a different set of parameter values for the choice target input, motion input, and sure target input that are adopted to the new experimental protocol of the Kiani-Shadlen experiment. Although the network was not originally designed for a confidence-estimation experiment, unexpectedly it can reproduce the behavioral and neurophysiological observations that are similar to those in the Kiani-Shadlen experiment [Figs. 1, *B* and *C*, and 2, *A* and *B*; Fig. 5, *B* and *C* in Kiani-Shadlen study (2009)]: *1*) neurons inside *T*_{A} and *T*_{B} display firing activities with small differences when *T*_{s} is chosen, while their firing activity is divergent when *T*_{s} is shown but waived [Fig. 2, *B* and *D*, *top*, vs. Fig. 2, *A* and *B*, in Kiani and Shadlen (2009)]; *2*) neurons inside *T*_{s} response field have weak and uninformative spontaneous activity before *T*_{s} onset and then exhibit a fast ramping followed by a decay after *T*_{s} onset [Fig. 2*D*, *bottom*, vs. Fig. 5, *B* and *C*, in Kiani and Shadlen (2009)]. In the model, we assumed that the input strength of each target goes to the same level attributable to the adaptation. For simplicity, we fixed *I*_{4}, while adjusting τ_{s} to match a majority of points on the performance curves of *P*_{sure} from Kiani and Shadlen (2009) [Fig. 3*B*, vs. Fig. 1, *B* and *C*, in Kiani and Shadlen (2009)].

We also studied the choice confidence in a reaction time (RT) version of the task. In this task, a network can generate choice at any time after the motion onset and, at the same time, report directly its choice confidence. We followed the same simulation protocols as those in the FD task without *T*_{s}, except that the motion input is terminated when one of the activity bumps crosses the decision threshold, 60 Hz for at least 50 ms. We measured the corresponding time, *t*_{r}, and calculated RT as *t*_{R} = *t*_{r} − *t*_{m} + 80, where 80 ms is the latency period for implementation of saccadic eye movement (Roitman and Shadlen 2002). In the FD version of the task, an initial choice is assumed to be made when one of the two competing neural pools reaches a decision threshold of 50 Hz for at least 50 ms after motion onset because the decision threshold of an FD task was experimentally observed lower than that of an RT task (Roitman and Shadlen 2002).

In FD tasks, we performed 1,500 trials at each motion strength and stimulus duration level, where *T*_{s} was not presented, and 3,500 trials at each motion strength and stimulus duration level, where *T*_{s} was presented. In Fig. 5, we simulated 1,500 trials for each condition. In RT task, we carried out 3,000 trials for each motion strength level. All the simulated behavioral data reported were computed using all trials for each simulation set. The integration method was a modified second-order Runge-Kutta algorithm with firing-time interpolation (Hansel et al. 1998) and a time step *dt* = 0.02 ms.

#### Measurements of activity trajectories.

We calculated the average response of the population of units associated with targets *T*_{A}, *T*_{B}, and *T*_{s}, namely *R*_{A}, *R*_{B}, and *R*_{s}, as the average firing activity of the neurons within 8.4° around each target with a time window of 100 ms preceding the time point (e.g., the moment of decision and onset of *T*_{s}) for *R*_{A}, *R*_{B}, and *R*_{s} in analysis, except for Figs. 2, *B* and *D*, and 7*A*. In Figs. 2, *B*, *D*, *E*, *F*, *G*, and *H*, and 7*A*, each trajectory or point represents the activity of a single neuron at each target. We applied a 100-ms Gaussian sliding window to smooth the peristimulus time histograms for the temporal evolutions of the firing rates of *R*_{A}, *R*_{B}, and *R*_{s} in Figs. 2, *B*, *D*, *E*, *F*, *G*, and *H*, and 7*A*.

#### Choice confidence assessment.

In the monkey experiment, as well as in our model simulations, the introduction of a sure target only serves as a probe examining the confidence of the system. That is to say, with carefully choosing the ratio of sure target reward to that of choice targets (i.e., *I*_{4} in our simulation), one can access the choice confidence across trials. In our simulation, the probability of opting for the sure target is bounded (Fig. 3*B*); it thus represents a good choice for estimating confidence. Furthermore, we will show later in results that the probability of choosing the sure target, *P*_{sure}, reflects the uncertainty of a choice in an opt-out task. Here, we defined choice confidence, *cc*, as the probability of waiving a sure target, i.e., *cc* = 1 − *P*_{sure}, using the trials in which *T*_{s} is presented (binned by 0.5 Hz; Fig. 6*A*, black circles). We also assumed that this probability can be predicted as a function of the differential activity |*R*_{A} − *R*_{B}|. We then performed the fit of a logistic function between |*R*_{A} − *R*_{B}| and *cc* = 1 − *P*_{sure}. At each differential activity level, *i*, *cc*^{i} = 1 − *P*_{sure} is computed as the mean of the decision result across the sample trials, *k*, <*s*^{i}_{k}> for *T*_{s}. The decision result, *s*, is a binary variable, i.e., *s* = 1, if *T*_{s} is waived; *s* = 0, if *T*_{s} is chosen. To perform the fit, we used the firing activity within a 100-ms time window before *T*_{s} onset in the FD task (Fig. 6*A*) for *R*_{A} and *R*_{B}:

*cc*^{i} = 1 − *P*_{sure}(|*R*^{i}_{A} − *R*^{i}_{B}|) = *b*_{1} + *a*[1 + exp(*k*|*R*^{i}_{A} − *R*^{i}_{B}| − *b*_{0})]^{−1}

Using all trials in FD task, we obtained *b*_{0} = 2.22 Hz; *b*_{1} = 1.01; *a* = −1.01; *k* = 0.089 (*R*^{2} = 0.98, Fig. 6*A*, red dashed line). Importantly, a real result here is to quantify confidence as a function of the neural activity. Confidence estimation is thus applicable to all trials, even without sure target presentation. We then used these estimated parameters to calculate *cc*^{i} for each sample trial in both FD and RT tasks, where *R*_{A} and *R*_{B} are the average firing rates within a 100-ms time window before *T*_{s} onset in FD task (Figs. 6, 7*A*, and 8*C*) and those before one of the bumps reaching a decision threshold in RT task (Fig. 9).

## RESULTS

We performed computer simulations of the Kiani-Shadlen task (Fig. 1*A*), using a neuronal circuit model (Furman and Wang 2008). In this model, the pyramidal cells are selective for motion direction as an analog stimulus feature and are uniformly distributed along a ring according to their preferred directions. Pyramidal cells are endowed with strong recurrent excitation, which is balanced by feedback inhibition mediated by interneurons (Fig. 1*B*). We assumed that the neural representation of motion stimulus in MT exhibits normalization (Heeger 1992; Treue et al. 2000) (Fig. 1*C*). The output from MT converges with other visual inputs such as choice targets to the decision circuit. Without loss of generality, we placed the choice targets *T*_{A} at 90°, *T*_{B} at 270°, and *T*_{s} at 180° (Fig. 1*D*). In a short delay after the target onset, a random-dots motion stimulus is presented, and, in our model, the network integrates the motion signal gradually over time. If a categorical choice is formed through attractor dynamics (Furman and Wang 2008; Wang 2002, 2008; Wong and Wang 2006), that choice is maintained in the form of a persistent activity pattern during a delay period. Because the network dynamics is stochastic, a decision may not be reached during stimulus presentation. Network activity continues to evolve during the delay through slow NMDA-mediated reverberation, and this process can be altered in the event that *T*_{s} is presented and this third option becomes available.

#### Network dynamics in an FD task with PDW at zero motion strength.

In experiments using single-unit recording, each neuron was recorded at a time, and its selectivity and dynamics were evaluated across trials, whereas, in our model, all neurons are monitored simultaneously in a single trial. At the population level, the ramping activity is demonstrated as the gradual development of a bell-shaped activity pattern (bump) around the direction of a selected target. The stimulated neural dynamics in Fig. 2 can be compared directly with single-neuron data from area LIP for a neuron with the preferred direction at *T*_{A}; *T*_{A} and *T*_{B} are equivalent to *T*_{in} and *T*_{opp} in the Kiani-Shadlen experiment (2009).

Figure 2*A* shows the spatiotemporal spiking activity of the network model in a trial without *T*_{s} presented. Although the input is identical to all the pyramidal cells at zero motion strength, the two activity bumps compete with each other through shared inhibitory feedback and stochastic recurrent dynamics. Eventually one bump ramps up, while the other one decays, leading to a categorical choice (Furman and Wang 2008; Wang 2002; Wong et al. 2007). The ramping-up bump is maintained by the persistent activity in delay (Compte et al. 2000). At the single-unit level, the firing activities of neurons at *T*_{A} and *T*_{B} (*R*_{A} and *R*_{B}, respectively) diverge over time after motion onset, leading to the choice of the network (*T*_{A}). The winning neural pool persists its firing activity until the end of delay (Fig. 2*B*).

In trials when *T*_{s} is presented (Fig. 2, *C* and *D*), again the attractor dynamics dictates the choice behavior of the network, and we identified a choice of *T*_{s} by the firing activity of neurons at *T*_{s} (*R*_{s}); the network selects *T*_{s} (Fig. 2*D*, dashed blue line) if *R*_{s} persists at a high rate, and it waives *T*_{s} (Fig. 2*D*, solid blue line) if *R*_{s} decays to a low rate. In the trial where *T*_{s} is waived, we observed the same divergence and persistent activity of *R*_{A} (choice) and *R*_{B} as that without *T*_{s} presented (Fig. 2, *C*, *top*, and *D*, solid lines); however, if the network selects *T*_{s}, *R*_{A} and *R*_{B} are indistinguishable at some intermediate rates without significant divergence (Fig. 2*C*, *bottom*, and *D*, dashed lines; *R*_{A} = 32.5 Hz, *R*_{B} = 33.6 Hz at *T*_{s} input onset), and then both decay to low rates as neurons around *T*_{s} win the competition. The average firing activities of *R*_{A}, *R*_{B}, and *R*_{s} across different motion strengths (100 trials for each condition) are shown in Fig. 2, *E* and *F*, which follow the same conventions as those in Fig. 2, *B* and *D*. All these are similar to the observed LIP neuronal data (Kiani and Shadlen 2009) from motion input onset time to the end of trial, which is crucial to capture the neural dynamics of choosing or waiving *T*_{s}, although a difference of neuronal activities could exist before motion input onset in our model compared to the experimental observations, which is not important to predict a choice of *T*_{A}, *T*_{B}, or *T*_{s}.

We further studied the neural dynamics that underlies a choice of the network among *T*_{A}, *T*_{B}, and *T*_{s}. Taking trials in Fig. 2*C*, for example, we visualized the neural trajectories in a 3D decision space (*R*_{A}, *R*_{B}, *R*_{s}) following the sequences marked in Fig. 2*D*. In the trial where *T*_{s} is waived (Fig. 2*G*, red line), the network first converges to a choice attractor (*T*_{A}), then leaves away from and returns to it again after the presentation of *T*_{s}; in the trial where *T*_{s} is selected (Fig. 2*G*, gray line), the network wanders around the diagonal line (*R*_{A} = *R*_{B}, *R*_{s} ≈ 0) and then converges to the attractor *T*_{s}. These trajectories in decision space imply that the presentation of *T*_{s} could induce a sure attractor, which behaves similarly to choice attractors; the network could thus act like a three-way competition after the presentation of *T*_{s}. To test this, we visualized more sampling trials at the same stimulus condition and explored the basin of attraction for each attractor (Fig. 2*H*). We found that, for trials choosing *T*_{A}, *T*_{B}, and *T*_{s} (Fig. 2*H*, red, green, and gray lines, respectively), the networks converge to the choice attractors *T*_{A} (near the *R*_{A} axis) and *T*_{B} (near the *R*_{B} axis) and the sure attractor *T*_{s} (near the *R*_{s} axis), respectively. The whole decision space is thereby separated into three attractor regions, and, because of this structure of decision space, the location of each neural trajectory at the moment of *T*_{s} onset (namely the initial location) can potentially predict the choice of a network (Kiani and Shadlen 2009). Particularly, if the initial location of a network is close to a choice attractor, it would eventually converge back to that choice attractor again after *T*_{s} onset, whereas, if its initial location is around the diagonal line, it more likely converges to a sure attractor. Notably, if the initial location of a network is between the diagonal line and a choice attractor, it could either continue converging to that choice attractor or change its mind to *T*_{s} (discussed later in Fig. 7). Therefore, our model demonstrates that a categorical choice of a network in this task could be generated by a three-way competition among attractors *T*_{A}, *T*_{B}, and *T*_{s}, which relies internally on the stochastic neural dynamics (Wang 2008).

#### Behavioral performance.

The performance of the model is quantified by the fraction of trials corresponding to a particular behavioral response. Figure 3 exhibits the probability of choosing *T*_{s} (*P*_{sure}) and accuracy (*P*_{correct}) for trials when *T*_{s} is not presented or *T*_{s} is shown but waived. At a fixed stimulus duration, our model shows that *P*_{sure} decreases while *P*_{correct} increases with the motion strength; *P*_{correct} improves in trials where *T*_{s} was shown but waived (Fig. 3*A*). Moreover, Fig. 3*B* shows that *P*_{sure} decreases with the stimulus duration, wherefore the network selects *T*_{s} more often for weaker motion strength or shorter duration; *P*_{correct} increases monotonically with stimulus duration for trials with or without *T*_{s} presented (Fig. 3*B*, *right*, solid and dashed lines, respectively). For a given motion strength and stimulus duration, *P*_{correct} is higher for trials where *T*_{s} is shown but waived than those without *T*_{s} presented, implying that *P*_{sure} is a probe of uncertainty (Kepecs and Mainen 2012; Whiteley and Sahani 2008). In conclusion, the model successfully reproduces the salient behavioral observations in the monkey experiment (Kiani and Shadlen 2009) (Fig. 3*C*).

#### Choice confidence as a logistic function of the differential activity.

Consistent with the neurophysiological observation (Kiani and Shadlen 2009), *R*_{A} and *R*_{B} undergo brief decreases after *T*_{s} onset in our model (Fig. 2*C*). This is because *T*_{s} stimulates neurons selective for the sure target, and their increased firing activity recruits more feedback inhibition that reduces *R*_{A} and *R*_{B} in a three-way competition (Fig. 2*H*). We therefore hypothesized that the network would opt to *T*_{s} if it has not converged to a stable attractor for *T*_{A} (*R*_{A} >> *R*_{B}) or *T*_{B} (*R*_{A} << *R*_{B}); the differential activity |*R*_{A} − *R*_{B}| at *T*_{s} onset determines *P*_{sure}.

We first observed that *R*_{A} and *R*_{B} could diverge in the late phase of delay. The long divergent time (>*T*_{s} onset) implies that there is a quasistable state at *R*_{A} = *R*_{B}, around which the network could wander, but eventually the network would escape from it and converge to a stable attractor, generating a choice. We next studied whether the network in a state around *R*_{A} = *R*_{B} would opt to *T*_{s} (e.g., Fig. 2*H*, gray lines). For individual sample trials, we visualized *R*_{A} and *R*_{B} against each other on a decision space. Figure 4, *A* and *B*, shows that the activity of the network falls down along a diagonal line and then separates into three groups for choices of *T*_{A}, *T*_{B}, and *T*_{s}. In trials where *T*_{s} is waived, the network converges to one of two stable attractors, *T*_{A} or *T*_{B} (Fig. 4, *A* and *B*, gray lines), while, in those where *T*_{s} is selected, the network walks randomly around the quasistable state at *R*_{A} = *R*_{B} ≈ 35 Hz (Fig. 4, *A* and *B*, black lines). In summary, once the network converges to a stable attractor before *T*_{s} onset, *T*_{s} is waived; if it wanders around *R*_{A} = *R*_{B}, the network is likely to opt to *T*_{s}.

The studies of the similar attractor models (Wang 2002; Wong and Wang 2006; Wong et al. 2007) showed that early divergence of *R*_{A} and *R*_{B} (bias to one attractor) determines the probability of choosing *T*_{A} and *T*_{B} as a function of the motion strength and stimulus duration. One can thus expect that early divergence would also result in a decrease of *P*_{sure} as a function of the motion strength and stimulus duration. To examine this, we investigated the distributions of *R*_{A} − *R*_{B} with different motion strengths at *T*_{s} input onset. Figure 4, *C* and *D*, shows that the percentage of the trials around *R*_{A} = *R*_{B} decreases with higher motion strength or longer stimulus duration, resulting in a decrease of *P*_{sure}.

Although the early divergence plays a dominant role in the network dynamics, network continues to evolve via NMDA-mediated reverberatory dynamics; the slow stochastic dynamics could thus drive the network away from *R*_{A} = *R*_{B} in the later phase of the delay. Consequently, *P*_{sure} depends on *T*_{s} onset time. Figure 5*A* displays the evolution of the distribution of *R*_{A} − *R*_{B} at different times from motion offset, demonstrating that the slow stochastic dynamics also plays an essential role in the behavior of the network. Across trials, our model predicts that *P*_{sure} decreases with longer *T*_{s} onset times (Fig. 5*B*) because more trials settle down to a stable attractor later in the delay, i.e., the percentage of trials with *R*_{A} = *R*_{B} decreases with *T*_{s} input onset time. Interestingly, we found that *P*_{correct} was nearly constant with different *T*_{s} input onset times (Fig. 5*C*). This happens because *P*_{correct} ≈ *r*_{AB}/(1 + *r*_{AB}), where *r*_{AB} is the ratio of the number of trials at attractor *T*_{A} (choice) to that at attractor *T*_{B} at *T*_{s} onset and is saturated at 575 ms after motion offset (Fig. 5*A*). In short, *P*_{sure} is directly related to the percentage of trials around *R*_{A} = *R*_{B}, whereas *P*_{correct} is associated with the number of trials at *T*_{A} and *T*_{B} at *T*_{s} onset. In this sense, there is a dissociation of confidence estimation from performance (Graziano and Sigman 2009; Graziano et al. 2010). Moreover, without *T*_{s} presented, the network continues to converge to *T*_{A} or *T*_{B} via stochastic dynamics, and the probability to one of them is biased and relies on *R*_{A} − *R*_{B} in the early phase of delay (Fig. 5*D*). In conclusion, we found that |*R*_{A} − *R*_{B}| at the moment of *T*_{s} input onset determines *P*_{sure} probabilistically and reflects a degree of the stability of a choice: if a categorical choice is achieved but with small |*R*_{A} − *R*_{B}|, it could be altered to *T*_{s}; whereas, if |*R*_{A} − *R*_{B}| is large, the choice of the network would not be changed by *T*_{s}.

Here we define choice confidence as a function of the instantaneous differential activity |*R*_{A} − *R*_{B}| at sure target input onset time for each trial *i*, i.e.,

*cc*^{i} = *f*(|*R*^{i}_{A} − *R*^{i}_{B}|).

In our model, |*R*_{A} − *R*_{B}| shows the position of a network in the (*R*_{A},*R*_{B}) plane (Fig. 2, *G* and *H*) related to choice attractors in the decision space, i.e., the larger |*R*_{A} − *R*_{B}| is, the closer a network is to a choice attractor *T*_{A} or *T*_{B}; *f*(·) is therefore required to be an increasing function. In the previous studies (Beck et al. 2008; Kepecs and Mainen 2012; Kepecs et al. 2008; Vickers 1979), functions *f*(·) were given in a variety of ways. One can picture that, as long as *f*(·) is a monotonic function, we can always equate *f*(·) from one model to another. Of note, our definition of the confidence stems from the structure of the attractor basin in decision space (Fig. 2, *G* and *H*), i.e., if a choice is confident, then it is easily resistant to the other external input such as a sure target, whereas confidence estimations from models like Beck et al. (2008) and Kepecs and Mainen (2012) are compared directly to log odds of a choice in Bayesian framework. In the studies of Beck et al. (2008), they found from both the experimental data and their model that log odds of choice at A, namely confidence across trials for choice A, is proportionate to <*R*_{A}> − <*R*_{B}>, where <·> is the average across trials. However, such a read-out of confidence would predict a strong correlation between confidence and performance on single trials, which is somewhat inconsistent with experimental observation of a broad performance variation in different confidence categories (Graziano et al. 2010; Juslin and Olsson 1997). Second, this “optimal decoder” <*R*_{A}> − <*R*_{B}> relies explicitly on the equal variance hypothesis for likelihood (Kepecs and Mainen 2012) or “left-right” symmetry of a linear decoder (Beck et al. 2008). It remains unclear what the biologically plausible mechanisms are to achieve such a fine-tuned neural circuit to compute confidence signals in these models.

In our model, confidence is defined using a monotonic increasing function of |*R*_{A} − *R*_{B}|. Particularly in an opt-out task, confidence can be probed by the probability that a choice stays in its attractor after presenting a sure target. If the choice is confident at |*R*_{A} − *R*_{B}|, then this probability, 1 − *P*_{sure}, is low. Using this probe, we found that choice confidence increases as a sigmoid function [i.e., function *f*(·)] of |*R*_{A} − *R*_{B}| (Fig. 6*A*). Next, we asked whether, across trials, our definition of confidence can also reflect probability of correct at each |*R*_{A} − *R*_{B}| level as those defined in Bayesian framework (Beck et al. 2008; Kepecs and Mainen 2012; Kepecs et al. 2008). This seems possible as indicated from Fig. 5*D*. Moreover, a detailed analysis should compare probability of correct choice and confidence simultaneously. We performed this analysis using trials without presenting *T*_{s}, where we computed both the performance and the confidence at the end of decision averaged across all stimulus conditions (Fig. 6*B*). Figure 6*B* demonstrates that confidence in our model increases monotonically with the performance. Importantly, our model indicates that confidence can be computed as a function of the instantaneous neural activities (like a population code; Beck et al. 2008) at any time in a decision circuit without explicit use of elapsed time for integration of the sample (Drugowitsch et al. 2012; Kiani and Shadlen 2009; Kiani et al. 2014; Moreno-Bote 2010). Therefore, although confidence in our model is not defined as a log odds function of the choice (Beck et al. 2008; Drugowitsch et al. 2012; Kepecs and Mainen 2012; Kepecs et al. 2008; Kiani and Shadlen 2009; Kiani et al. 2014), confidence can be a good measurement of the subjective correct rate across trials. Importantly, on single trials, choice confidence in our model is dissociable from performance (Graziano et al. 2010), whereas Bayesian models would predict a strong correlation.

Despite the similarity of *f*(·), choice confidence in our model is, however, conceptually distinct from those from Bayesian decision models because our definition of confidence fundamentally comes from the structure of the attractor basin in the decision space. Therefore, our model predicts that confidence would be different when estimated at the different times after motion offsets (Fig. 5), and that would be nearly the same in a Bayesian model (Beck et al. 2008; Drugowitsch et al. 2012; Kiani and Shadlen 2009; Moreno-Bote 2010). To test this, we estimated choice confidence using neural activities *R*_{A} and *R*_{B} in trials without *T*_{s} presented at different times after motion offsets. We first estimated the choice confidence at 575 ms after motion input offset (Fig. 6*C*; compared directly with Fig. 3*B*, *left*), where the distribution of *R*_{A} − *R*_{B} is still evolving, namely the confidence estimation after a short delay (Fig. 5*A*). One can thus expect an increase of choice confidence in trials with longer stimulus durations (Fig. 6*C*), according to the variation of the bimodal distribution of *R*_{A} − *R*_{B} at different stimulus durations (Fig. 4, *C* and *D*). We next estimated the choice confidence at 1,550 ms after motion input onset, i.e., the same time of a trial, where the internal noise is nearly identical at different stimulus conditions and the strength of the input dominates the choice confidence of the network. In this case, one would expect that confidence should increase as a function with the motion strength and stimulus duration for a noiseless integrator (Beck et al. 2008), unless it is bounded (Drugowitsch et al. 2012; Kiani and Shadlen 2009). While in our model, the attractor dynamics implies that the bimodal distribution of *R*_{A} − *R*_{B} depends dominantly on the early divergence (Fig. 4, *C* and *D*). As a result, Fig. 6*D* shows that all confidences saturate at stimulus duration >400 ms, suggesting that the early evidence has the greatest effect on confidence estimation. Of note, the saturation time prolongs with lower motion strength.

#### Low confidence results in changes of mind to T_{s}.

The whole dynamic space, *R*_{A}-over-*R*_{B} decision space, can be divided into three regions: choice attractor regions (*R*_{A} >> *R*_{B} or *R*_{A} << *R*_{B}) and an unstable region in between them (Wang 2008). In the previous study, we focused on the trials along the diagonal line (*R*_{A} = *R*_{B}), where a choice of network remains undecided before the presentation of *T*_{s}. We then investigated the dynamics of networks that are between the diagonal lines (*R*_{A} = *R*_{B}) and a choice attractor (*R*_{A} >> *R*_{B} or *R*_{A} << *R*_{B}) on *R*_{A}-over-*R*_{B} decision space preceding the presentation of *T*_{s}, where a trial could be still in the unstable region (and thus could go to the sure attractor after *T*_{s} onset) or in a stable attractor region (where the network trajectory would stay in the same choice attractor even after *T*_{s} onset). For these trials, we could define an initial choice of the network by its nearby choice attractor *T*_{A} (*T*_{B}, respectively), where *R*_{A} (*R*_{B}, respectively) fires above a decision threshold (>50 Hz). Particularly, we explored under which condition the network would more likely continue converging to the attractor of its initial choice, or shift to the sure attractor, when *T*_{s} is offered.

Figure 7*A* compares the neural activity in trials with low- and high-confidence initial choices, the analysis of which is on single trials and is missing in Kiani and Shadlen (2009). In low-confidence trials (Fig. 7*A*, *top*), only one of the activity bumps fires above the decision threshold, but the other remains similar (|*R*_{A} − *R*_{B}| is small). After *T*_{s} onset, both *R*_{A} and *R*_{B} decay to a low level, while *R*_{s} grows to a high level and *T*_{s} is chosen. By contrast, in high-confidence trials (Fig. 7*A*, *bottom*), when one of the firing rates reaches a steady state, the other is much smaller (|*R*_{A} − *R*_{B}| is large). Neurons activated by *T*_{s} are suppressed, and *T*_{s} is waived. In the latter case, the activity of the winning neural pool exhibits a brief dip upon *T*_{s} input onset and then ramps up again to its steady state.

Across trials, the probability of changes of mind to *T*_{s} is negatively correlated with choice confidence, i.e., the network exhibits low confidence in trials at low motion strength (Fig. 6*C*) and high probability of changes of mind to *T*_{s} (Fig. 7*B*, *left*). To further test whether the network bases the probability of changes of mind to *T*_{s} on its performance and confidence, we categorized the trials with initial choices, where either *R*_{A} or *R*_{B} reaches a decision threshold, 50 Hz (if both of them do not reach the decision threshold, we considered the choice remaining undecided at *T*_{s} onset), into correct and error groups and found that network changes its choice to *T*_{s} more often in error trials. Furthermore, the probability of choosing *T*_{s} in correct (error, respectively) choice decreases (increases, respectively) with the motion strength (Fig. 7*B*, *right*). This finding is reminiscent of the experimental observation that, in a decision making task with a delayed reward, animals moved back to self-restart port more often when the task became more difficult or the confidence was low (Kepecs et al. 2008).

In conclusion, we identified two possibilities for choosing *T*_{s}; either an initial choice was not made (along the diagonal line; Fig. 2), or it was made with a low confidence (between the diagonal line and choice attractors; Fig. 7*A*, *top*). For the latter case, |*R*_{A} − *R*_{B}| reveals the confidence about an initial choice; low confidence of a choice is likely to result in changes of mind to *T*_{s}.

#### A sure target as a probe about the confidence of the system.

The introduction of a sure target plays a role of probing the confidence of the system. Specifically, in the monkey experiment, the physical luminance of the sure target was the same as the choice targets. Monkeys were trained to understand what the sure target meant behaviorally, which depended on the amount of reward by choosing it. Therefore, in our model, the amplitude of the sure target input (*I*_{4} in our model) does not correspond to its physical properties but is related to the behavioral significance of the sure target that a monkey learned as the amount of reward he receives by choosing the sure target. One can imagine that, if choosing the sure target yields a negligible (significant, respectively) amount of reward, monkeys would never (always, respectively) have learned to choose it. To test this, we studied the effect of *T*_{s} input strength, *I*_{4}, on the behavioral performance at a fixed motion strength level (i.e., 12.8%). We found that *P*_{sure} increases as a function of *T*_{s} input strength (Fig. 8*A*). When *T*_{s} input strength is low (e.g., *I*_{4} = 192 pA), *T*_{s} is always waived; when *T*_{s} input strength is high (*I*_{4} = 288 pA), *T*_{s} is mostly chosen, as stimulus duration is short. Moreover, in the trials where *T*_{s} is shown but waived, our simulation predicts an increase of correct rate at high *T*_{s} input strength (Fig. 8*B*). At the network level, these observations in Fig. 8, *A* and *B*, still follow a three-way competition among *R*_{A}, *R*_{B}, and *R*_{S}, e.g., when input of the sure target is weak (strong, respectively), it always behaves like a loser (winner, respectively). Last, we examined whether, in a range of *T*_{s} input strengths (a selected range of amount of *T*_{s} rewards), a sure target can serve as a probe about the confidence of the system, when the network applies the attractor dynamics. We assessed the choice confidence as a function of |*R*_{A} − *R*_{B}| at the moment of *T*_{s} input onset for different choices of *T*_{s} input strengths. Figure 8*C* shows that, on average, choice confidence is identical for different *T*_{s} input strengths (and increases as a function of stimulus durations), and *P*_{sure} decreases as a linear function of choice confidence for a broad range of *T*_{s} input strengths, i.e., in this range, 216 pA <*I*_{4} < 264 pA, a sure target in our model can be considered as a probe about the confidence of the system. Therefore, a sure target is only a probe, and the confidence measure is valid even without it.

#### Assessment of choice confidence in an RT task.

In our model, confidence can be read out at any time and increases as a function of stimulus duration in an FD task. One may thus argue that the network would exhibit high confidence despite the task difficulty if it freely controls the viewing duration of the stimulus. However, classical literature about confidence in cognitive psychology (Vickers 1979) emphasizes an inverse relationship between confidence and response time, which can be potentially tested in an RT version of discrimination task [developed previously by Furman and Wang (2008)] with direct assessment of choice confidence. This distinguishable difference between confidence estimation in FD vs. RT task in fact comes from two distinct processes; whereas longer viewing time in an FD task enables more integration of evidence (confidence thus increases with motion viewing time), a longer RT means a higher task difficulty in an RT task (confidence thus decreases instead with motion viewing time). We thus want to further test whether our model can nicely explain such a contrasting observation. To do this, we designed an RT version of discrimination task with direct assessment of choice confidence (Fig. 9*A*); the network integrates the motion input until the neurons selective for one of two alternatives fire above a decision threshold and reports the confidence as the function of the instantaneous |*R*_{A} − *R*_{B}| (Fig. 6*A*) at the moment of choice [a similar human behavioral experiment is performed and reported by Kiani et al. (2014) recently].

Our model exhibits the typical psychometric and chronometric curves of a two-alternative discrimination task (Churchland et al. 2008; Roitman and Shadlen 2002), i.e., *P*_{correct} increases, whereas RT decreases with the motion strength (Fig. 9, *B* and *C*). Importantly, weaker motion strengths are associated with longer RTs, where |*R*_{A} − *R*_{B}| will be less at longer RTs. Choice confidence thus increases with the motion strength [Fig. 9*D*; see also Fig. 5 in Beck et al. (2008)] and is positively correlated with the behavioral performance across trials (Barthelmé and Mamassian 2010) (data not explicitly shown). We also found that choice confidence decreases as an inverse function of RT (Fig. 9*E*), which agrees broadly with the human behavioral observations (Vickers 1979). Although an erroneous choice could be associated with high confidence (Graziano and Sigman 2009), the average |*R*_{A} − *R*_{B}| across trials is higher in correct trials than that in error ones (Wang 2002). Therefore, in our model, confidence increases (decreases, respectively) with motion strength in correct (error, respectively) trials (Fig. 9*F*), consistent with human studies (Pierrel and Murray 1963).

Moreover, we studied correlation between the choice confidence and decision accuracy. Figure 9, *B* and *D*, implies that choice confidence is positively correlated with behavioral performance across trials. Although confidence in our model does not directly represent a subjective estimation of performance (like that in Beck et al. 2008; Drugowitsch et al. 2012; Kiani and Shadlen 2009), one can estimate the subjective performance from choice confidence using a monotonic function, *g*(·), in Fig. 6*B*. We can thus compare directly our confidence score with performance to study the “hard-easy” effect (Juslin and Olsson 1997). Here we defined underconfidence score as the difference between the choice confidence and accuracy, *cc* − *P*_{correct} [one can also use *g*(*cc*) − *P*_{correct}], and a hard-easy effect is the observation that the underconfidence score decreases as a function of task difficulties, i.e., in the easy (difficult, respectively) trials, the report is more likely to be overconfident (underconfident, respectively) *cc* − *P*_{correct} > 0 (*cc* − *P*_{correct} < 0, respectively). Figure 9, *G* and *H*, shows the variation of underconfidence scores as a function of choice confidence for the FD task with a delay of 627 ms and RT task, respectively; both display the hard-easy effect in the reports. Of note, these results still hold true when comparing estimated subjective performance from choice confidence with the behavioral performance using *g*(*cc*) − *P*_{correct}. Such a hard-easy effect in our model mainly stems from sampling of stochastic neural dynamics; sampling duration thus influences the underconfidence score in our model. When the sampling duration is short, the network behaves with more overconfidence. To test this, we compared the scores in the FD and RT tasks, where the average sampling durations in RT tasks are longer than those in FD tasks at low motion strength (from 0 to 6.4%). Consequently, the network exhibits overconfidence more often at low motion strengths in FD task.

## DISCUSSION

We have shown that a biologically plausible spiking network model can account for salient physiological and behavioral data from an experiment designed to study confidence (Kiani and Shadlen 2009), and in our model internal stochasticity plays an essential role of choice confidence (see also Whiteley and Sahani 2008). Specifically, at the moment of choice, our model simultaneously generates a neural signal for confidence. Confidence can be estimated as a function of the differential activity of the competing neural populations, |*R*_{A} − *R*_{B}|. Compared with Bayesian inference models, in our model, there is no explicit representation of probabilities such as likelihood or posterior function. Indeed, all computations are carried out by the fluctuating neural network dynamics. Therefore, confidence estimation itself is simply a quantity that stochastically varies over time and from trial to trial under the same stimulus condition.

Our identification of a confidence signal, |*R*_{A} − *R*_{B}|, agrees with the idea that, as a metacognitive process, confidence is estimated directly on a decision process (Graziano and Sigman 2009; Graziano et al. 2010; Middlebrooks and Sommer 2011, 2012). At the same time, choice confidence is also dissociable from whether the decision is correct or wrong in a single trial, as illustrated by high-confidence error trials (Fig. 7*A*, *bottom*, *right*). In line with our model, the EEG data from (Graziano et al. 2010) showed that, at the neural level, choice confidence could be dissociated from performance. Such dissociation is naturally explained by attractor dynamics, which could yield the same magnitude of the differential activity |*R*_{A} − *R*_{B}|, hence the same confidence rating in correct and error trials. It is worth noting that *1*) *R*_{A} and *R*_{B} represent the choices of a decision (not necessary to be a directional decision-making process); *2*) confidence estimation does not depend on a specific choice of the decision (i.e., it does not exclusively rely on the activity of the winner bump, or the losing bump) but a differential activity between choices. In this case, one would expect that a sure target is chosen when |*R*_{A} − *R*_{B}| is small or the downstream neuronal activity is weak and that a nonsure target is chosen when |*R*_{A} − *R*_{B}| is large or the downstream neuronal activity is strong. This prediction from our model is consistent with the observations in Komura et al. (2013), wherefore the finding of pulvinar neuronal activity (Komura et al. 2013) could be an example of |*R*_{A} − *R*_{B}| in the downstream read-out circuit of confidence.

In our model, fast early divergence, i.e., the difference of early buildup rates between *R*_{A} and *R*_{B}, has a predominant effect on the choice and confidence. This is manifested in the dependence of the choice confidence on the stimulus duration, which saturates quickly for sufficiently long stimuli (Fig. 6*D*; Wong et al. 2007). By contrast, in DDM, sensory evidence contributes equally in time to confidence estimation. Future experiments are needed to assess this different characteristic of the attractor network model vs. DDM. Furthermore, the two competing neural pools could also diverge slowly later in a trial. In our model, persistent activity during the delay not only maintains working memory but also continues to slowly integrate signals from memory (Curtis and Lee 2010). This provides a neural mechanism for postdecision sampling (Resulaj et al. 2009). For instance, Fig. 7*B* shows that the probability of switching from an initial decision to *T*_{s} is higher in error trials, in agreement with behavioral observation in a rat experiment (Kepecs et al. 2008). This finding also sheds insights into the phenomenon of changes of mind, which may result from the instability (low confidence) of a choice (see also Albantakis and Deco 2011).

Of note, in the monkey experiment, as well as in our model simulations, the introduction of a sure target only serves as a probe about the confidence of the system (Fig. 8). The probability of opting for the sure target is bounded (Figs. 3 and 8*C*), so it represents a good choice for estimating confidence. The real result, we emphasize here, is to quantify confidence as a function of the neural activity |*R*_{A} − *R*_{B}|. Confidence thus quantified should be applicable to all trials, even without sure target presentation. Furthermore, in the analysis of Kiani and Shadlen (2009), they also found that the probability of opting for the sure target can be predicted using either *f*(|*R*_{A} − <*R*_{A}>|) or *f*(|*R*_{B} − <*R*_{B}>|). Nevertheless, *f*(|*R*_{A} − <*R*_{A}>|) or *f*(|*R*_{B} − <*R*_{B}>|) is not a good measure of confidence for an RT task, for which either *R*_{A} or *R*_{B} is assumed to reach a fixed threshold at the moment of the choice; therefore, *f*(|*R*_{A} − <*R*_{A}>|) or *f*(|*R*_{B} − <*R*_{B}>|) would always be a fixed value [*f*(|threshold − average|)] rather than a graded quantity that varies from trial to trial (Kiani et al. 2014).

#### Comparison with existing models.

Computational schemes have been proposed for the study of confidence (Kepecs and Mainen 2012; Kepecs et al. 2008; Kiani and Shadlen 2009; Moreno-Bote 2010; Ratcliff and Starns 2009; Rolls et al. 2010a, 2010b; Vickers 1979). These models can be classified into Bayesian inference models and neural network models.

In Bayesian inference models, one can either compute confidence based on a single decision variable (Drugowitsch et al. 2012; Kiani and Shadlen 2009) or an optimal population code (Beck et al. 2008). Kiani and Shadlen (2009) proposed that confidence could be defined in terms of the log posterior ratio for the two choices given the position of a decision variable and elapsed time at decision, using DDM. This looks promising, yet it remains unclear what is a direct representation of a decision variable exclusively for a choice. Moreover, for the RT version of the task, this kind of model implies that the position of a decision variable at decision time would be a deterministic function of RT (either a constant or a time-varying function like that in Drugowitsch et al. 2012); one can thus find that confidence would also decrease deterministically as a monotonic function of RT (Drugowitsch et al. 2012; Kiani and Shadlen 2009; Volkmann 1934) on single trials. This idea, however, failed to explain the widely overlapped RT distributions in different confidence categories (Ratcliff and Starns 2009). Such a strong correlation between confidence and RT or performance can be eliminated through a two-stage DDM (Pleskac and Busemeyer 2010), where additional process for confidence is required. Nevertheless, in our model, performance, RT, and confidence are naturally dissociated with each other on single trials. Importantly, in a classic DDM model, sensory evidence contributes equally in time to confidence estimation, whereas, in our model, confidence estimation would more be dominated by the early sensory evidence. Last, in the Kiani and Shadlen model, choosing *T*_{s} uses a hard thresholding process and has little to do with neural activity at *T*_{s} response field, whereas, in our model, it generates from the same sampling of stochastic neural dynamics as the other choices (as indicated by data in Fig. 5, Kiani and Shadlen 2009).

On the other hand, the optimal population code model (Beck et al. 2008) posits that confidence could be estimated as the instantaneous differential activity, |*R*_{A} − *R*_{B}|, without explicit use of RT as our model. A notable difference between our model and theirs is that the optimal population code model requires LIP neural circuit to be a fine-tuned noiseless integrator. This can be easily tested experimentally because our model predicts that confidence estimation would differ at different times in the delay, whereas their model would expect that to be constant. Generally, these Bayesian inference models (Beck et al. 2008; Kiani and Shadlen 2009) assert that confidence must be based on explicit neural representation of probability functions, such as likelihood at any moment in time and in single trials. Our model demonstrates an alternative perspective. Whereas probability representations may be a perfectly valid mathematical description of the aggregated statistics across trials, they should not be confused with what actually happens in single trials, which is stochastic neural dynamics.

Insabato et al. (2010) and Rolls et al. (2010a, 2010b) extended the model of Wang (2002) to account for the confidence estimation and its behavioral readout. Insabato et al. (2010) argued that confidence can be read out as a function of *R*_{A} + *R*_{B}, and Rolls et al. (2010a, 2010b) claimed that it can be further approximated as a function of the neural activity of the winning pool. All these models showed some consistencies with the existing data. However, as discussed in our model, neither of them can exclusively demonstrate the position of a neural trajectory related to the choice attractor and thus the choice confidence at any time during a decision. These results seem only true at the moment when a decision is made exactly around a choice attractor in the decision space, where |*R*_{A} − *R*_{B}| ≃ |*R*_{A} + *R*_{B}| ≃ max(*R*_{A},*R*_{B}) because min(*R*_{A},*R*_{B}) << max(*R*_{A},*R*_{B}). Therefore, these models would fail to predict confidence using a fixed decision boundary or capture the relationship between neural activities in LIP with *P*_{sure} or high-confidence errors in single trials. Alternatively, our model does not require a time-varying decision threshold, estimates confidence simply as a function of instantaneous |*R*_{A} − *R*_{B}| at the moment of choice on single trials, and can correctly reproduce the salient behavioral relationships between confidence, RT, and performance on single trials and those across trials.

Confidence rating is important for monitoring cognition when there is uncertainty, and two types of uncertainty should be distinguished, namely Brunswikian (external) uncertainty originating from incomplete states of knowledge (noisy or ambiguous sensory data) and Thurstonian (internal) uncertainty attributable to variations intrinsic to the brain (Juslin and Olsson 1997). The noise level in a decision circuit has only recently begun to be examined experimentally (Brunton et al. 2013). Our work provides a computational framework to detect these two effects using a spiking-neuron circuit. Our model can also be extended in several important ways. It still remains an open question how confidence estimation, as a sigmoid function of the differential activity in downstream neural circuits, can be read out for a direct report and to guide future behavior. In fact, confidence is commonly assessed without a verbal report using a two-stage PDW task; subjects perform a first-order discrimination task and then make a high-low bet on the outcome of the decision (Middlebrooks and Sommer 2011, 2012; Smith 2009) (see also Kiani et al. 2011), where the probability for a high bet is considered as a readout of confidence estimation. A plausible neural circuit for explicit representation and memory of a confidence signal is needed for the two-stage PDW (Komura et al. 2013; Middlebrooks and Sommer 2011, 2012) and should be examined in the future. A biologically plausible neural circuit to computing |*R*_{A} − *R*_{B}| involves neurons in pulvinar (Komura et al. 2013), where the neurons fires highly in nonsure-target trials and low in sure-target trials. Moreover, Kiani et al. (2014) recently found that confidence could also decrease in error trials with the difficulty of task, which poses a challenge to our model prediction (Fig. 9*F*). One possible direction in the future is to understand the mechanism of error trial. Nevertheless, what is robust about the prediction in our model compared with their observation is that the difference of the confidence between correct and error decreases with the difficulty of task (Lak et al. 2014). Second, for the sake of simplicity, we assumed that the amount of *T*_{s} reward is encoded as the onset strength of its target input, which mimics the firing activity of midbrain dopamine neurons in response to the targets with different amounts of reward (Tobler et al. 2005). Our model predicts that both *P*_{sure} and *P*_{correct} increase with the reward of *T*_{s} (data not shown). This then brings up two inquiries for future studies: *1*) what a reasonable amount of *T*_{s} reward used to measure confidence in a PDW task would be (Dienes and Seth 2010; Fleming and Dolan 2010; Persaud et al. 2007) and *2*) how the amount of *T*_{s} reward used is learned through neural dynamics and applied to the decision circuit (Soltani and Wang 2006). Moreover, one can extend our model to investigate confidence signals for multiple-choice decision tasks and effects of microstimulation on confidence (Fetsch et al. 2014). Specifically, one can incorporate known effects of microstimulation on MT inputs in our model to perform the experiment of Fetsch et al. (2014) using computer simulation and then test its effects on confidence. Finally, confidence may be represented in a distributed network in the brain (Del Cul et al. 2009); the dynamical nature and computational principle remains to be elucidated in future research. In conclusion, we found it remarkable that a previously established model of decision making (Furman and Wang 2008) naturally accounts for all the salient behavioral and neurophysiological observations of the Kiani-Shadlen experiment (2009). Furthermore, it reproduces the observation that confidence decreases with response time in an RT version of the task. The model also offers testable predictions about the changes of mind and, unexpectedly, the hard-easy effect observed in human studies, which naturally emerges from the model. Taken together, our work establishes that a dynamical system of stochastic neural population can underlie even the seemingly abstract metacognitive concept of confidence.

## GRANTS

This work was supported by NIH grant R01-MH062349.

## DISCLOSURES

No conflicts of interest, financial or otherwise, are declared by the authors.

## AUTHOR CONTRIBUTIONS

Z.W. and X.-J.W. conception and design of research; Z.W. performed experiments; Z.W. analyzed data; Z.W. and X.-J.W. interpreted results of experiments; Z.W. prepared figures; Z.W. and X.-J.W. drafted manuscript; Z.W. and X.-J.W. edited and revised manuscript; Z.W. and X.-J.W. approved final version of manuscript.

## ACKNOWLEDGMENTS

We thank M. Furman for help with the computer code (Furman and Wang 2008) and D. Wang for help with simulations. Model simulations were carried out at Beijing Normal University.

- Copyright © 2015 the American Physiological Society