The inferior olive is the sole source of the climbing fiber system; one of the two major afferent systems of the cerebellum, however, its exact role remains unknown. A longstanding hypothesis is that the inferior olive with its unique intrinsic rhythmic firing properties mediates motor timing. However, direct evidence linking the inferior olive to timing behavior has been difficult to demonstrate in animal or human studies likely due to the inhibition of inferior olive responses by self-produced movement. Here, we used event-related functional magnetic resonance imaging (fMRI) and a perceptual task that dissociates the temporal from non-temporal attributes of sensory input. Subjects were asked to attend to rhythmically occurring identical visual stimuli and to detect a change in their timing, spatial orientation or color. Inferior olive activation was seen only when perceiving a change in stimulus timing. These results are consistent with animal studies demonstrating that the inferior olive is especially sensitive to "unexpected" sensory events and further provide evidence supporting the specificity of the inferior olive response to stimulus timing. The results are consistent with the view that the inferior olive and the climbing fiber system mediate the encoding of temporal information required for both motor and non-motor cognitive processes.