Our ability to process temporal frequency information by touch underlies our capacity to perceive and discriminate surface textures. Auditory signals, which also provide extensive temporal frequency information, can systematically alter the perception of vibrations on the hand. How auditory signals shape tactile processing is unclear; perceptual interactions between contemporaneous sounds and vibrations are consistent with multiple neural mechanisms. Here we used a crossmodal adaptation paradigm, which separated auditory and tactile stimulation in time, to test the hypothesis that tactile frequency perception depends on neural circuits that also process auditory frequency. We reasoned that auditory adaptation effects would transfer to touch only if signals from both senses converge on common representations. We found that auditory adaptation can improve tactile frequency discrimination thresholds. This occurred only when adaptor and test frequencies overlapped. In contrast, auditory adaptation did not influence tactile intensity judgments. Thus auditory adaptation enhances touch in a frequency- and feature-specific manner. A simple network model in which tactile frequency information is decoded from sensory neurons that are susceptible to auditory adaptation recapitulates these behavioral results. Our results imply that the neural circuits supporting tactile frequency perception also process auditory signals. This finding is consistent with the notion of supramodal operators performing canonical operations, like temporal frequency processing, regardless of input modality.
NEW & NOTEWORTHY Auditory signals can influence the tactile perception of temporal frequency. Multiple neural mechanisms could account for the perceptual interactions between contemporaneous auditory and tactile signals. Using a crossmodal adaptation paradigm, we found that auditory adaptation causes frequency- and feature-specific improvements in tactile perception. This crossmodal transfer of aftereffects between audition and touch implies that tactile frequency perception relies on neural circuits that also process auditory frequency.
we are exquisitely sensitive to mechanical oscillations in our environment. Indeed, using our sense of touch, we are able to detect and perceive vibrations up to nearly 1 kHz. Specialized receptors in the skin transduce vibrations into neural signals that are transmitted from the peripheral afferent systems to the brain, where they are elaborated in extensive networks of somatosensory brain regions (Saal and Bensmaia 2014). The processing of vibration signals is thought to underlie our ability to discriminate texture information by touch (Manfredi et al. 2014) and to sense the environment through handheld tools (Johnson 2001).
In a variety of contexts, touch can be strongly influenced by audition. Sounds can alter the detection of vibrations (Ro et al. 2009; Wilson et al. 2010), the perception of vibration frequency (Yau et al. 2009), and even the subjective experience of textures (Guest et al. 2002; Jousmäki and Hari 1998). These perceptual interactions can be highly specific in the frequency domain and reciprocal domain (Yau et al. 2010), so the neural mechanisms mediating tactile frequency perception may not be completely separable from their auditory counterparts. Indeed, a number of sensory brain regions have been shown to respond to both senses (Butler et al. 2012; Caetano and Jousmäki 2006; Foxe et al. 2002; Fu et al. 2003; Kayser et al. 2005; Lemus et al. 2010; Nordmark et al. 2012; Schroeder et al. 2001; Schürmann et al. 2006). However, the neural mechanisms by which audition influences touch are unclear. Interactions could occur through neural populations that represent tactile frequency information and also explicitly process auditory frequency signals. Alternatively, audition could influence touch through the interplay between neurons that separately represent tactile and auditory frequency in parallel. Previous studies have mostly tested for interactions between contemporaneous sounds and vibrations (although see Wilson et al. 2009, in which nonoverlapping stimuli were used in a detection paradigm), which can be explained by either mechanism. Crucially, characterizing the influence of sounds on vibrations that do not overlap in time could arbitrate between the two mechanisms. Thus establishing the time horizon for audio-tactile perceptual interactions could clarify the nature of the neural mechanisms mediating the influence of audition on touch.
Here, we tested the hypothesis that auditory inputs have access to the neural circuits supporting tactile frequency perception even when the two modalities are stimulated at different times. We exploited adaptation, a general phenomenon in which neurons change their response properties depending on temporal context (Solomon and Kohn 2014), to test our hypothesis. Because adaptive neural changes can lead to systematic perceptual changes, adaptation has been used extensively as a tool for relating perception to neural function. We used a crossmodal adaptation paradigm and reasoned that perceptual aftereffects induced by auditory adaptation should transfer to touch only if tactile frequency representations are accessible to offline auditory signals. In separate sessions (Fig. 1A), participants performed a tactile frequency discrimination task with and without initial adaptation to auditory band-passed noise (BPN). By manipulating the spectral content of the BPN adaptors, we established the frequency specificity of auditory adaptation effects on tactile frequency perception. In two control experiments, we determined that auditory adaptation did not affect the tactile perception of stimulus intensity. We implemented a simple neural network model to identify potential mechanisms by which auditory adaptation could shape tactile frequency perception.
MATERIALS AND METHODS
Twenty subjects (15 females; mean age ± SE: 22.60 ± 0.67 yr old) participated in the frequency discrimination experiment (experiment 1). Ten subjects (5 females; 23.00 ± 1.44 yr old) participated in the intensity discrimination experiment using the 200-Hz standard and the 200-Hz comparison stimuli (experiment 2). Ten subjects (6 females; 25.00 ± 0.82 yr old) participated in the intensity discrimination experiment using the 200-Hz standard and the 350-Hz comparison stimuli (experiment 3). One subject participated in all three experiments. Three additional subjects participated in both experiments 1 and 3. One subject participated in experiments 2 and 3. All but one participant were right-handed according to the Edinburgh Handedness Inventory (mean scores, experiment 1: 0.69 ± 0.06; experiment 2: 0.58 ± 0.09; experiment 3: 0.74 ± 0.06). Individuals with known auditory or tactile deficits were excluded from study participation. All of the experimental procedures were approved by the Baylor College of Medicine Institutional Review Board. All participants gave their written, informed consent and were paid for their participation.
Vibrotactile stimuli were delivered along the axis perpendicular to the skin surface by a plastic stylus (8-mm diameter) mounted on a shaker motor (Fig. 1; K2007E01, SmartShaker; The Modal Shop, Cincinnati, OH). The probe tip was initially indented into the skin by 1 mm to ensure contact with the skin throughout the stimulus presentation. The shaker motor was equipped with an accelerometer (type 8702B50M1; Kistler Instrument, Amherst, NY) with a dynamic range of ± 50 g. The accelerometer output was amplified and conditioned using a piezotron coupler (type 5134A; Kistler Instrument). The output of the piezotron coupler was digitized (PCI-6229; National Instruments, Austin, TX; sampling rate = 20 kHz) and read into a computer.
We used two approaches to attenuate the sounds produced by the shaker motor. First, we had participants wear noise-attenuating earmuffs (Peltor H10A Optime 105 Earmuff; 3M, St. Paul, MN). Second, we designed a sound-attenuation chamber to house the shaker motor (Fig. 1C). The walls of the sound-attenuation chamber consisted of three layers: a hard polyurethane board (84775K23; McMaster-Carr, Robbinsville, NJ), 1-inch-thick polyurethane acoustical foam insulation (5692T49; McMaster-Carr), and a 3-inch-thick egg carton polyurethane foam sheet (9710T46; McMaster-Carr). The shaker motor was mounted to an adjustable stage (UMR8.51; Newport, Irvine, CA) that was supported by a custom-built aluminum frame. The participant placed his or her left hand through an entry hole (lined with foam) and rested his or her index finger on a support platform mounted directly below the shaker motor and contact probe. In separate experiments, we verified that the noise-attenuating earmuffs and chamber prevented participants from relying on acoustic cues from the shaker motor to perform the tactile frequency discrimination task; performance fell to chance level when participants performed the task in the absence of finger contact with the stimulator.
Auditory stimuli consisting of BPN were used as adaptors. To assess the frequency dependence of crossmodal adaptation effects, we tested two auditory noise adaptors. The BPN stimuli were centered on either 200 or 400 Hz and had 100-Hz bandwidths. The amplitudes of the BPN200Hz and BPN400Hz stimuli were 53 and 60 dB SPL, respectively, which were subjectively matched in preliminary experiments. Stimuli were generated digitally and converted to analog signals using a digital-to-analog card (PCI-6229; National Instruments; sampling rate = 20 kHz). The analog signals were amplified (PTA2 stereo power amplifier; Pyle Audio, Brooklyn, NY) before being delivered binaurally via noise-cancelling in-ear headphones (ATH-ANC23; Audio-Technica U.S., Stow, OH). Participants wore noise-attenuating earmuffs over the in-ear headphones.
Participants were recruited separately for the frequency (experiment 1) and intensity (experiments 2 and 3) discrimination experiments. In experiments 1 and 3, each participant completed three sessions (Fig. 1A): baseline, adaptation with BPN200Hz, and adaptation with BPN400Hz. Sessions occurred on separate days (mean intersession interval: 5.8 ± 1.8 days), and the order of all three sessions was counterbalanced across subjects. The baseline session involved only the tactile discrimination task. The adaptation sessions involved auditory noise adaptation in addition to the tactile discrimination task. Experiment 2 only comprised the baseline and BPN200Hz adaptation sessions (mean intersession interval: 4.0 ± 0.5 days).
Before the start of each session, participants went through a training period of the frequency (or intensity) discrimination task to ensure that they understood the directions and could perform the task. Each session comprised five task blocks. In the adaptation sessions, an initial 3-min adaptation period preceded each tactile discrimination task block (Fig. 1A). During the adaptation period, subjects were exposed to continuous auditory noise stimulation. During the task block, participants performed 32 trials of a perceptual discrimination task in a two-alternative forced choice (2AFC) design. On each trial (Fig. 1B), participants were exposed to a 6-s reinforcement or “top-up” adapting noise stimulus before the presentation of the tactile test stimuli. These brief “top-up” adaptation intervals were designed to maintain the state of adaptation induced by the initial 3-min adaptation period. A 3- to 5-min break separated the conclusion of a task block and the start of the next adaptation period.
Frequency Discrimination Task
Participants discriminated vibration frequency in a 2AFC paradigm. On each trial, a pair of vibrotactile stimuli was delivered sequentially to the participant’s left index finger; his or her task was to report which of the two vibrations was higher in frequency. Stimuli were presented for 1 s and were separated by a 1-s interstimulus interval. One interval always contained a 200-Hz vibration (standard stimulus); the frequency of the vibration presented during the other interval (comparison stimulus) ranged from 100 to 300 Hz. The frequency of the comparison stimulus and the stimulus interval in which it was presented were randomized across trials. Stimulus amplitudes were equated for perceived intensity (see Equating Tactile Stimulus Intensity) to ensure that participants could only discriminate vibrations by relying on frequency information. The subjectively matched amplitudes of the vibration frequencies (100, 140, 160, 180, 200, 220, 240, 260, and 300 Hz) were 7.86, 3.12, 2.53, 2.32, 2.28, 2.32, 2.41, 2.52, and 2.78 μm, respectively. All vibration amplitudes are given as peak amplitudes. Tactile stimuli delivered to the finger at these amplitudes, although clearly suprathreshold, still fall below levels required for detection via bone conduction (Dirks et al. 1976; Yau et al. 2010). To further ensure that participants could not rely on stimulus intensity to perform frequency discrimination, the actual stimulus amplitudes tested during the experiments were randomly jittered (maximum jitter was ± 15% of the subjectively matched amplitude). Twenty observations were obtained for each comparison stimulus within each session. No feedback was provided.
Equating Tactile Stimulus Intensity
In preliminary experiments, we equated the perceived intensity of vibrations at different frequencies using an adaptive 2AFC tracking procedure. This procedure yielded an iso-intensity curve from which we could determine subjectively matched amplitudes for the test stimuli used in the main experiment. On each trial, participants were presented sequentially with two vibrations (1-s duration; 1-s intersession interval). One stimulus (the standard) was always a 300-Hz, 1.25-μm vibration; the other stimulus (the comparison) was a vibration whose frequency ranged from 100 to 550 Hz (100, 140, 180, 200, 220, 250, 260, 350, 400, 450, 500, and 550 Hz). Participants reported which stimulus was more intense. If the participant judged the standard as more intense, the amplitude of the comparison stimulus increased on the following trial. Conversely, if the participant judged the comparison as more intense, the comparison amplitude was reduced on the following trial. Amplitude changes on trials earlier in the test run were initially large (0.5 μm), but changes after the comparison amplitude reversed six times were finer (0.1 μm). The test run concluded when the change in the amplitude of the comparison stimulus reversed a total of 12 times. We then computed the geometric mean of the comparison stimulus amplitudes on the last 10 trials of the run. The adaptive procedure was repeated five times for each comparison stimulus using different starting amplitude values. The final amplitude for each comparison frequency was calculated by averaging over the five repeats. Five participants performed these preliminary experiments, and a single iso-intensity curve was fitted to the data aggregated over all subjects. The stimulus amplitudes in experiment 1 were determined as two times the values in the fitted iso-intensity curve. Note that, because of intersubject variability and intrasubject noise, the iso-intensity curve did not pass exactly through 1.25 μm at 300 Hz, so the nominal amplitude tested at 300 Hz after doubling was 2.78 μm rather than 2.50 μm.
Intensity Discrimination Task
Participants discriminated vibration intensity in a 2AFC paradigm. On each trial, participants were asked to determine which of two sequentially presented vibrations (1-s duration; 1-s intersession interval) was perceived as more intense. In experiment 2, the standard stimulus was a 200-Hz vibration delivered at 2.25 μm. The frequency of the comparison stimulus was 200 Hz, and amplitudes ranged from 1.0 to 3.5 μm. In this experiment, we only tested participants in the baseline and BPN200Hz adaptation sessions. Experiment 3 also required participants to discriminate vibration intensity, but this experiment differed from experiment 2 in two ways: 1) the frequencies of the standard and comparison stimuli were different, and 2) participants were tested in all three sessions (baseline, BPN200Hz, BPN400Hz). In experiment 3, the standard stimulus was a 200-Hz vibration delivered at 3μm. The frequency of the comparison stimulus was 350 Hz, and the amplitudes ranged from 1 to 6 μm (1.0, 1.7, 2.5, 3.1, 4.0, 4.9, 5.7, and 6.0 μm). Twenty observations were obtained for each comparison stimulus within each session.
Generalized linear mixed models.
For each experiment, we fitted generalized linear mixed models (GLMM) (Moscatelli et al. 2012) to data aggregated over all participants. These models, which have become increasingly popular in psychophysical analyses (Moscatelli et al. 2015; Moscatelli et al. 2016; Tomassini et al. 2014), explicitly account for random effects (subject-specific variability) as well as fixed effects (experimental conditions). With the use of the GLMM approach, hypotheses on fixed effects can be tested by comparing the performance of competing models that differ in their assumptions about the experimental conditions. Our analysis determined whether tactile frequency sensitivity differed significantly across baseline and adaptation sessions.
We fitted a model that allowed tactile frequency sensitivity to differ across sessions (adaptation-dependent model): where is the probit transform (i.e., the inverse of the cumulative Gaussian function) of the probability that the ith subject judged the comparison stimulus, , on the jth trial to be higher in frequency than the standard stimulus, and are categorical predictors coding for the adaptation conditions, and and are random-effect predictors. The fixed-effect parameters, θ*, represent the effects of the experimental conditions (vibration and adaptation frequencies). With the coding of the adaptation conditions using dummy variables, the control condition without adaptation corresponds to the baseline in this model. Accordingly, θ0 and θ1 represent the intercept and slope of the response function in the baseline condition, respectively.
We also fitted a model that assumed the same frequency sensitivity across sessions (adaptation-invariant model): where the probit transform of the response probabilities is simply a linear combination of fixed- and random-effect predictors that apply equally to baseline and adaptation conditions.
We compared the adaptation-dependent and adaptation-invariant models using both a likelihood ratio test and Akaike information criterion. We calculated the Wald (z) statistic to test hypotheses regarding the fixed-effect slope parameters corresponding to frequency sensitivity (i.e., the just-noticeable difference, JND). We used a bootstrap procedure (Moscatelli et al. 2012) to generate confidence intervals for estimates of JND and the point of subjective equality (PSE). We applied the same analysis approach in testing hypotheses about the influence of auditory adaptation on tactile intensity perception. GLMM fitting, statistical testing, and bootstrapping were performed in R (https://www.r-project.org/; http://mixedpsychophysics.wordpress.com).
Parameter as outcome measures.
We supplemented the GLMM analysis with a conventional analysis based on group-level statistical tests of psychophysical parameters estimated separately for each participant. To quantify each participant’s ability to discriminate tactile frequency, we fitted the participant’s performance data with a Gaussian cumulative distribution function (cdf): where p(fc > fs) is the proportion of trials a comparison stimulus with frequency fc was judged to be higher in frequency than the standard stimulus (fs), μ and σ are free parameters corresponding to estimates of the participant’s PSE and JND, respectively, and erf(x) is the error function of x. The PSE is a measure of bias and indicates the comparison frequency perceived as equal to the standard frequency. The JND is a measure of sensitivity that is defined as the standard deviation of the Gaussian fit, which corresponds to 84% performance. Participants’ ability to discriminate vibration intensity in experiments 2 and 3 was similarly quantified using the Gaussian cdf.
In the group-level analysis for each experiment we determined whether the average JND and PSE estimates differed significantly as a function of adaptation condition (baseline, BPN200Hz, BPN400Hz). We conducted a repeated-measures ANOVA with adaptation condition as the within-subjects factor. We excluded a participant’s data from these analyses if his/her psychometric function goodness of fit (i.e., variance explained) for any adaptation condition fell outside of the interquartile range (IQR) plus or minus 2 times the IQR determined from all conditions across all participants. On this basis, data from two participants were excluded from experiment 1 analysis. No data were excluded from experiments 2 and 3. Psychometric function fitting and statistical testing were conducted using Matlab (MathWorks, Natick, MA). If the repeated-measures ANOVA indicated a significant main effect of adaptation condition, we performed post hoc paired t-tests contrasting adaptation conditions. We also performed post hoc binomial tests to assess whether the number of participants showing effects in the direction indicated by the paired t-tests exceeded chance levels.
Participants reliably discriminated vibration frequency in baseline and adaptation sessions. Figure 2A shows representative performance data depicting the likelihood a vibration of a given comparison frequency was perceived as higher than the 200-Hz standard. We tested the effects of adaptation in our participant sample using generalized linear-mixed models (materials and methods). To test whether auditory adaptation modulates tactile frequency sensitivity, we compared a GLMM that allowed frequency sensitivity to differ across adaptation conditions (adaptation-dependent model) to a GLMM that assumed identical frequency sensitivity across adaptation conditions (adaptation-invariant model). The adaptation-dependent model significantly outperformed the adaptation-invariant model based on a likelihood ratio test [ = 18.587, P < 0.001] and was the preferred model according to Akaike information criterion (AIC, adaptation-dependent model: 1907.8, adaptation-invariant model: 1918.4). GLMM analysis also revealed significantly greater sensitivity in BPN200Hz sessions compared with baseline sessions (Wald test, z = 3.9, P < 0.0002) but no differences between baseline and BPN400Hz sessions (z = 0.6, P = 0.55). Sensitivity in the BPN200Hz sessions was also significantly greater than sensitivity in the BPN400Hz sessions (z = 3.3, P < 0.002).
We confirmed these results in a separate analysis in which we fitted psychometric functions to each participant’s data (Fig. 2A; mean r2 = 0.94) before conducting group-level tests on JND estimates (Fig. 2B). JND estimates from each session (means ± SE; baseline: 51.5 ± 6.4 Hz, BPN200Hz: 39.6 ± 2.9 Hz, BPN400Hz: 46.7 ± 2.2 Hz) differed significantly according to a repeated-measures ANOVA (rmANOVA; F2,34 = 3.22, P = 0.05). This modest statistical result may be due to the fact that this analysis, unlike the analysis based on the GLMM, ignores subject-specific standard error. Consistent with the GLMM analyses, JND estimates with BPN200Hz adaptation were significantly lower compared with baseline [paired t-test, t(17) = 2.36, P < 0.05, uncorrected] and compared with estimates with BPN400Hz adaptation [paired t-test, t(17) = 2.3, P < 0.05, uncorrected]. JND estimates with BPN400Hz adaptation did not differ from baseline [paired t-test, t(17) = 0.84, P = 0.41]. Additionally, the number of participants whose discrimination thresholds improved with BPN200Hz adaptation compared with baseline or BPN400Hz adaptation was significant according to binomial tests (Fig. 2C; 13/18; P < 0.05). In contrast to the clear frequency-specific effects on JND, estimates of PSE (Fig. 2, B and D; baseline: 195.9 ± 3.1 Hz, BPN200Hz: 199.5 ± 2.6 Hz, BPN400Hz: 199.1 ± 2.1 Hz) did not differ across adaptation conditions according to an rmANOVA (F2,34 = 0.83, P = 0.44) and GLMM analysis. Thus analyses based on GLMM and rmANOVA yield consistent results; auditory adaptation induces frequency-specific alterations to tactile frequency sensitivity but not bias.
To test whether the influence of auditory adaptation on touch is restricted to the frequency domain or whether adaptation influences touch generally, we assessed the effect of adaptation on tactile intensity perception. Tactile intensity processing is susceptible to adaptation when tactile adaptors are used (Berglund and Berglund 1970; Goble and Hollins 1993); however, we predicted that tactile intensity processing would remain unchanged after auditory adaptation because tactile intensity judgments are impervious to concurrent auditory stimulation (Yau et al. 2009). In experiment 2, participants judged which of two 200-Hz vibrations on each trial was perceived as more intense (materials and methods). Given the pattern observed with frequency perception, we contrasted performance in the BPN200Hz session with baseline performance (Fig. 3, A and B). GLMM analysis failed to reveal significant differences in sensitivity [likelihood ratio test, = 1.51, P = 0.47] and PSE. Furthermore, neither JND (Fig. 3, B and C; baseline: 0.47 ± 0.06 μm, BPN200Hz: 0.43 ± 0.04 μm) nor PSE (Fig. 3, B and D; baseline: 2.30 ± 0.03 μm, BPN200Hz: 2.28 ± 0.02 μm) values differed between baseline and BPN200Hz sessions [paired t-test; JND: t(9) = 0.65, P = 0.53; PSE: t(9) = 0.50, P = 0.63].
The negative result in experiment 2 may have been due to the fact that the standard and comparison tactile stimuli on each trial were matched in frequency; adaptation effects could have been obscured if adaptation equally influenced the perception of the tactile stimuli presented during the two trial intervals. To address this possibility, we determined whether participants’ ability to compare the intensity of 200- and 350-Hz vibrations could be shaped by exposure to auditory adaptors in experiment 3 (Fig. 4). GLMM analysis again failed to reveal significant adaptation-related differences in intensity sensitivity [likelihood ratio test, = 5.24, P = 0.26] and PSE. Similarly, neither JND (Fig. 4, B and C; baseline: 0.85 ± 0.12 μm, BPN200Hz: 1.15 ± 0.16 μm, BPN400Hz: 0.98 ± 0.08 μm) nor PSE (Fig. 4, B and D; baseline: 3.92 ± 0.28 μm, BPN200Hz: 3.71 ± 0.27 μm, BPN400Hz: 3.74 ± 0.39 μm) estimates differed significantly between sessions according to the analysis of parameters estimated for each participant (rmANOVA; JND: F2,18 = 2.66, P = 0.1; PSE: F2,18 = 0.82, P = 0.45). Our collective results over all adaptation experiments, then, reveal that the tactile perception of frequency, but not intensity, is modified by auditory adaptation.
By what mechanism could auditory adaptation affect neural circuits to improve performance on a tactile frequency discrimination task? As a first step to address this question, we adopted a maximum-likelihood decoding approach in constructing simple models to relate neural responses to frequency perception (Fig. 5A; Appendix). By assuming that the model sensory neurons charged with encoding frequency are susceptible to adaptation and accessible to both auditory and tactile inputs, we tested distinct mechanisms by which auditory adaptation could alter the sensory neuron responses to produce the observed tactile perceptual changes. Adaptation has been shown to alter the gain, width, and peak location of neural tuning curves exclusively or in combination (Dragoi et al. 2000; Schwartz et al. 2007; Solomon and Kohn 2014). For simplicity’s sake, we assumed that adaptation could produce bidirectional changes to the response magnitude of a sensory neuron, frequency selectivity, or frequency preference (Fig. 5B). Additionally, we tied the magnitude of adaptation-driven changes in a given neuron to its responsiveness to the adapting stimulus; neurons with tuning preferences closer to the adaptor exhibited larger changes. We fitted each adaptation model to the group-averaged performance data from the frequency discrimination experiments (Table 1). Only the model based on tuning shifts clearly reproduced the observed JND changes (Fig. 6A). Specifically, repulsive shifts in the tuning functions away from the adapting stimulus led to improved frequency discrimination performance and smaller JND values with BPN200Hz adaptation. All of the models predicted an absence of adaptation effects on PSE (Fig. 6B). Thus a simple model comprising supramodal neurons whose tuning functions shift with adaptation recapitulates the observed behavioral patterns.
We perceive temporal frequency information by audition and touch. These senses display highly specific interactions in the frequency domain when sounds and vibrations occur together. Our results demonstrate that frequency-specific interactions between audition and touch are possible even when the auditory and tactile stimuli are separated in time. The finding that auditory adaptation alters subsequent tactile frequency perception, but not intensity perception, supports the hypothesis that tactile frequency perception relies on neural circuits that also process auditory signals. A simple model that assumes the existence of supramodal frequency-tuned neurons provides a potential mechanism for how auditory adaptation could lead to frequency-specific improvements in tactile frequency discrimination performance.
Prolonged exposure to auditory noise enhances tactile frequency sensitivity in a frequency-dependent manner, without concomitant influences on PSE (Fig. 2). This pattern is unlikely to be due to attentional factors or cognitive biases for many reasons. First, the fact that the auditory adaptors and the tactile test stimuli were not delivered concurrently in our experiment obviates concerns that interactions between audition and touch arise from a division of attention over both senses. Previous work showing that audio-tactile frequency interactions occur with asynchronous but temporally overlapping stimulation patterns (Yau et al. 2009; Yau et al. 2010) is also consistent with the notion that audio-tactile interplay does not require attentional binding, which would be cued by synchronicity. Second, the influence of auditory adaptation on touch is restricted to the frequency domain and depends on the similarity between the adaptor’s spectral content and the tactile test frequencies. Such specificity indicates that tactile sensitivity is not simply altered through some general modality-reweighting mechanism triggered by exposure to auditory noise. Third, the adaptors consisted of band-passed noise stimulation that did not evoke clear pitch percepts by themselves. Because the adaptors did not evoke pitch experiences, the adaptation effects are unlikely to be operating on higher-order pitch representations that may be easier to hold in memory either implicitly or explicitly. Notably, auditory noise distractors experienced concurrently with vibrations also systematically influence tactile perception in a frequency-specific fashion (Yau et al. 2009), so neither the online nor offline influences of audition on touch require auditory pitch generation. Last, because response times were statistically equivalent across sessions (rmANOVA; main and interaction effects including adaptation as a factor, P > 0.12), participants did not appear to adopt different response strategies or performance criteria across the experimental sessions. Rather than engaging touch through higher-order attentional or decisional processing mechanisms, auditory signals may instead shape tactile perception by altering the activity in networks that represent tactile frequency.
If auditory signals impinge on tactile frequency representations, where might these interactions occur? In the somatosensory cortical system of nonhuman primates, information about low-frequency “flutter” stimuli can be encoded by both temporal (Mountcastle et al. 1990) and rate codes (Salinas et al. 2000), but only the latter are highly predictive of trialwise perceptual decisions (Salinas et al. 2000). In contrast, the spectral content of high-frequency vibrations such as those used in our study only appear to be encoded in spike timing (Harvey et al. 2013); evidence for explicit, rate-based tuning for vibration frequency remains elusive. However, multiple regions in classically defined auditory cortex contain neurons that exhibit explicit frequency tuning (Bendor and Wang 2005). These areas, which clearly support auditory frequency processing, are obvious candidate sites of auditory and tactile convergence. In fact, audition and touch activate multiple overlapping regions in the auditory cortices of humans (Butler et al. 2012; Caetano and Jousmäki 2006; Foxe et al. 2002; Nordmark et al. 2012; Schürmann et al. 2006) and nonhuman primates (Fu et al. 2003; Kayser et al. 2005; Lemus et al. 2010; Schroeder et al. 2001). Noninvasive brain stimulation targeting human auditory cortex can modulate tactile perception (Bolognini et al. 2010; Yau et al. 2014). Auditory association areas, namely the caudomedial and caudolateral belt areas (area CM and CL, respectively), are particularly interesting because unit activity in these regions reflects auditory and tactile processing even when these senses are stimulated in isolation (Fu et al. 2003; Schroeder et al. 2001). Furthermore, neurons residing in auditory belt regions exhibit broad tuning for temporal frequencies below 1 kHz (Recanzone 2000; Recanzone et al. 2000), matching the preferences of our model sensory neurons (Table 1). Substantial anatomical projections connect auditory regions directly and indirectly to somatosensory areas (Cappe et al. 2012; Hackett et al. 2007; Ro et al. 2013; Smiley and Falchier 2009), which may also exhibit bimodal responses (Beauchamp and Ro 2008). Because of these physiological and anatomical characteristics, the densely interconnected regions within perisylvian cortex could be sites of auditory and tactile signal convergence, but this remains to be tested. Additionally, regions in frontal cortex (Spitzer and Blankenburg 2012; Vergara et al. 2016) could support processing of both auditory and tactile signals.
To better understand how auditory adaptation could improve tactile frequency sensitivity, we constructed a simple network model that comprised a neuron pool responsive to both audition and touch. We found that a feedforward architecture that strategically pools the noisy activity of a small sample of frequency-tuned neurons is able to reproduce human tactile frequency discrimination performance. Models including different numbers of neurons produced similar results. Because adaptation can alter the response properties of individual neurons in a variety of ways, we tested multiple potential adaptation mechanisms. We found that modeling adaptation effects as repulsive shifts in neural tuning functions away from the adaptor frequencies yielded the best fit to our behavioral data. The failure of the gain and selectivity models may be due to the relatively broad tuning of the model sensory neurons; different assumptions about network architecture and information read-out could yield different conclusions regarding the role of adaptive changes in response gain and selectivity. Adaptation has been shown to cause repulsive shifts in the tuning functions of single neurons (Dragoi et al. 2000; Felsen et al. 2002; Müller et al. 1999). Why neurons alter their response properties is beyond the scope of this study (although see Schwartz et al. 2007), but, given that adaptation resulted in improved discrimination thresholds, our results are consistent with the general notion that adaptation serves to maximize the response range and sensitivity of an organism given the statistics of its sensory environment (Solomon and Kohn 2014). Our modeling results also highlight a key point regarding encoding and decoding for perception. We could only reliably reproduce the behavioral results in our in silico experiments by confining adaptation effects to the sensory neurons; the pooling rules (i.e., the model weights) used to generate the full likelihood function were identical in the pre- and postadaptation states. Model performance was significantly impaired when we also allowed the pooling rules to update according to the postadaptation properties of the sensory neurons. These results imply that the downstream decoding mechanisms operate blindly with respect to the tuning changes of the sensory neurons. In other words, the decision network performs exactly the same computations regardless of the adaptation state of the sensory neurons. This pattern is consistent with previous computational work on adaptation in the context of visual decision making (Seriès et al. 2009), which has also posited that such decoding ambiguity (Fairhall et al. 2001) or coding catastrophe (Schwartz et al. 2007) may convey functional advantages. Although our model is clearly an oversimplification of actual neural networks, it provides an instructive demonstration of how the perceptual consequences of crossmodal adaptation can be linked to neural changes in a perceptual decision network. Additional work is required to clarify the complex dependencies between network architecture, adaptive neural changes, and perceptual outcomes.
Adaptation has long been exploited in experimental paradigms to assess the specificity of perceptual and neural processes. Adaptation paradigms have revealed distinct perceptual channels in touch (Goble and Hollins 1993; Goble and Hollins 1994; Hollins et al. 1990; Tommerdahl et al. 2005) and audition (Alais et al. 2015; Parra and Pearlmutter 2007; Zwicker 1964). We used a crossmodal adaptation paradigm to test the modality specificity of the neural mechanisms that support touch. We observed modest but significant improvements in frequency discrimination thresholds; the average JND estimate with BPN200Hz adaptation was ~23% lower than its baseline counterpart. Threshold improvements of comparable magnitude have been reported with visual motion direction adaptation (Seriès et al. 2009). Vibrotactile adaptation alone results in substantial improvements in tactile discrimination thresholds (~35% threshold changes with respect to 200 Hz) (Tommerdahl et al. 2005); however; these larger effects could be due to adaptive changes occurring in peripheral and central mechanisms, whereas crossmodal adaptive changes are likely limited to central processes. We did not test whether tactile adaptation can similarly alter auditory perception because tactile capture of auditory frequency perception is relatively weak (Yau et al. 2010). Presumably, online and offline tactile influences on auditory perception may be more obvious with unreliable or ambiguous auditory signals. We also did not test manipulations of a number of stimulus parameters spanning different scales of timing, level, and spectral content. These parameters should presumably impact adaptation effects, so further studies are required to test whether the crossmodal transfer of adaptation effects in the temporal frequency domain are limited to the conditions we tested. Notably, a recent study found that auditory and visual adaptation cause frequency-specific enhancement of tactile discrimination of flutter frequencies (Badde et al. 2016), which suggests that frequency representations may be shared by the sensory modalities over multiple ranges.
Our results show that tactile frequency perception, but not intensity perception, is shaped by prior exposure to auditory noise. Crossmodal adaptation in the frequency domain strongly implies the existence of shared representations of auditory and tactile frequency. These representations would be consistent with the existence of supramodal mechanisms that perform canonical computations independent of sensory modality. Crossmodal adaptation occurs for other low-level sensory domains like motion (Konkle et al. 2009), rate (Levitan et al. 2015), and shape (Tal and Amedi 2009) and in higher-order processes like identity and emotion perception (Watson et al. 2014). Conceivably, the ubiquity of multisensory responses across cortex (Ghazanfar and Schroeder 2006) reflects the prevalence of supramodal mechanisms.
This work was supported by an Alfred P. Sloan Research Fellowship (J. Yau), R01NS097462 (J. Yau), and NSF IGERT fellowship (L. Crommett).
No conflicts of interest, financial or otherwise, are declared by the authors.
L.E.C, A.P.B, and J.M.Y. conception and design of research; L.E.C. performed experiments; L.E.C. and A.P.B. analyzed data; L.E.C. and J.M.Y. interpreted results of experiments; L.E.C. prepared figures; L.E.C. and J.M.Y. drafted manuscript; L.E.C., A.P.B., and J.M.Y. edited and revised manuscript; L.E.C., A.P.B., and J.M.Y. approved final version of manuscript.
We thank the Yau Laboratory members for thoughtful discussions, W. Nash, W. Quinlain, and V. Careagas for assistance with the noise attenuation box, J. Killebrew for technical assistance, M. Rahman, A. Rosenberg, M. Cai, and R. Brockman for feedback on the modeling work, A. Moscatelli for assistance with the GLMM, and J. Craig, S. Bensmaia, and C. Connor for helpful comments on earlier drafts.
Current affiliation for A. Perez-Bellido: Radboud University, Donders Institute for Brain, Cognition and Behavior, Nijmegen, The Netherlands.
APPENDIX: MODELING FREQUENCY DISCRIMINATION AND ADAPTATION
Frequency Encoding and Decoding
We adopted a maximum-likelihood decoding approach in constructing simple models to relate neural responses to stimulus perception (Jazayeri and Movshon 2006). Our model consisted of two layers. One layer contains a population of frequency-tuned sensory neurons whose firing statistics are described by a Poisson process. We made a set of simplifying assumptions about the tuning of the sensory neurons in the preadaptation state (see Parameter Estimation). The second layer contained a population of neurons that pooled the responses of the sensory neurons in a weighted fashion, determined by the tuning preferences and firing statistics of the sensory neurons, to represent stimulus likelihoods, L. These describe the likelihood that a stimulus with frequency ν was presented given the pooled activity of the sensory neurons. The full likelihood function, then, represents the likelihood of all possible frequencies given the pattern of noisy activity in a broadly tuned sensory neuron population.
For any given stimulus with frequency ν, the overall log likelihood function logL(ν) is given by: where Ri is the response rate of the ith neuron, in a pool of N neurons, whose frequency tuning is described by Gi. We assumed that the frequency tuning of each neuron was described by a Gaussian function, so where ai is the maximum response rate of the neuron, μi is its preferred frequency, and σi is its tuning width. The contribution of each sensory neuron to the measurement of the log likelihood function of stimulus ν is determined by its firing rate and its tuning function at ν. In simulating performance on the tactile frequency discrimination task, the “perceived” frequency during a given trial interval was determined as the peak of the full log likelihood function generated for that interval. We report the results of simulations based on a sensory neuron pool consisting of 10 neurons. A pool consisting of 50 neurons yielded similar results.
Frequency Adaptation in the Sensory Neuron Population
We assumed that the extent to which the tuning function of a neuron changed following adaptation depended on its responsiveness to the adapting band-passed noise (BPN) stimulus. To determine the magnitude of the adaptation effect, Fi, on the ith neuron, we first summed its responses to frequency components comprising the BPN adaptor with center frequency, CF: where le and ue are the lower and upper edges of the BPN spectrum, respectively. With this approach, adaptive change in each neuron is scaled by its tuning. To establish the relative amount of adaptive change for each neuron with respect to the rest of the neuron population, we normalized the Fiof each neuron by the maximum value in the population:
Although we evaluated responses at three component frequencies for the analyses included in this report, in separate analyses we determined adaptation magnitudes by evaluating more BPN component frequencies and achieved similar results.
Modeling Adaptation Changes to Response Magnitude, Selectivity, and Preference
We tested three unique models that limited adaptive changes either to the gain, width, or peak location of the tuning function of the sensory neurons.
The gain model assumes that adaptation can increase or decrease the maximum response rate of a given neuron with respect to its preadapted maximum response rate. We defined a single parameter, Cmax, which expresses the direction and magnitude of the gain change in the neuron most affected by adaptation. The maximum rate for the ith neuron following adaptation, , is given by: Cmax can range from −1 to 1, so ranges from 0 to
The selectivity model assumes that adaptation can increase or decrease the tuning width of a given neuron with respect to its preadapted tuning width. We defined a single parameter, Cmax, which expresses the direction and magnitude of the tuning width change in the neuron most affected by adaptation. The tuning width for the ith neuron following adaptation, , is given by: Cmax can range from −1 to 1, so ranges from 0 to .
The shift model assumes that the peak location of the tuning function is attracted to or repulsed from the center frequency, CF, of the adapting stimulus. Because the CF of the adaptor defines the limit to the attractive shifts of tuning functions, we tested attractive and repulsive shifts in separate adaptation models. For each model, we defined a single parameter, Cmax, which expresses the magnitude of the tuning peak shift in the neuron most affected by adaptation.
In the repulsion model, the peak location of the tuning function shifts downward if is lower than the CF of the adaptor. Conversely, the peak shifts upward if is higher than the CF of the adaptor. The peak location following adaptation is given by:
In the attraction model, the peak location of the tuning function shifts upward if is lower than the adaptor’s CF. Conversely, the peak shifts downward if is higher than the adaptor’s CF. The peak location following adaptation is given by: where . In this definition of adaptation, tuning peaks cannot shift beyond the CF of the adaptor.
We assumed fixed relationships in the preadaptation response properties of the sensory neurons in the encoding layer; 1) the tuning peak locations (μi) (i.e., the preferred frequencies) were uniformly distributed in the frequency domain (50–700 Hz); 2) the tuning widths (σi) varied (inverse) logarithmically as a function of preferred frequency; and 3) the response gain (ai), indicating the maximum response level for each neuron, was identical for all neurons.
With these assumptions, each adaptation model included just three free parameters (a, σ, Cmax) corresponding to the baseline response gain, baseline neural tuning width, and maximum adaptation strength, respectively. We separately fitted each adaptation model to the group-averaged performance data from the baseline and adaptation sessions using least-squares optimization to minimize the sum of squared differences between the observed and predicted performance rates. Model fitting was performed using Matlab.
- Copyright © 2017 the American Physiological Society