Neurons in the auditory midbrain are sensitive to differences in the timing of sounds at the two ears-an important sound localization cue. We used broadband noise stimuli to investigate the interaural-delay sensitivity of low-frequency neurons in two midbrain nuclei: the inferior colliculus (IC) and the dorsal nucleus of the lateral lemniscus (DNLL). Noise delay functions showed asymmetries not predicted from a linear dependence on interaural correlation: a stretching along the firing rate dimension (rate asymmetry), and a skewing along the interaural-delay dimension (delay asymmetry). These asymmetries were produced by an envelope-sensitive component to the response, which could not entirely be accounted for by monaural or binaural nonlinearities. Instead it indicated an enhancement of envelope sensitivity at or after the level of the superior olivary complex. In IC, the skew-like asymmetry was consistent with intermediate-type responses produced by the convergence of ipsilateral peak-type inputs and contralateral trough-type inputs. This suggests a stereotyped pattern of input to the IC. In the course of this analysis, we were also able to determine the contribution of time and phase components to neurons' internal delays. These findings have important consequences for the neural representation of interaural timing differences and interaural correlation-cues critical to the perception of acoustic space.