Selecting a stimulus as the target for a goal-directed movement involves inhibiting other competing possible responses. Both target and distractor stimuli activate populations of neurons in topographic oculomotor maps such as the superior colliculus. Local inhibitory interconnections between these populations ensure only one saccade target is selected. Suppressing saccades to distractors may additionally involve inhibiting corresponding map regions to bias the local competition. Behavioural evidence of these inhibitory processes comes from the effects of distractors on oculomotor and manual trajectories. Individual saccades may initially deviate either towards or away from a distractor, but the source of this variability has not been investigated. Here we investigate the relation between distractor-related deviation of trajectory and saccade latency. Targets were presented with, or without, distractors, and the deviation of saccade trajectories due to the presence of distractor was measured. A fixation gap paradigm was used to manipulate latency independently of the influence of competing distractors. Shorter-latency saccades deviated towards distractors and longer-latency saccades deviated away from distractors. The transition between deviation towards or away from distractors occurred at a saccade latency of around 200 ms. This shows that the time course of the inhibitory process involved in distractor suppression are relatively slow.