To perform eye or hand movements towards a relevant location, the brain must translate sensory input into motor output. Recent studies revealed segregation between circuits for translating visual information into saccadic or manual movements, but less is known about translation of tactile information into such movements. Using human fMRI in a delay paradigm, we factorially crossed sensory modality (vision or touch) and motor effector (eyes or hands) for lateralised movements (gaze shifts to left or right, or pressing a left or right button with the corresponding left or right hand located there). We investigated activity in the delay-period between stimulation and response, asking whether the currently relevant side (left or right) during the delay was encoded according to sensory modality, upcoming motor response, or some interactive combination of these. Delay activity mainly reflected the motor response subsequently required. Irrespective of visual or tactile input, we found sustained activity in posterior partial cortex, frontal-eye field and contralateral visual cortex, when subjects would later make an eye-movement. For delays prior to manual button-press response, activity increased in contralateral pre-central regions, again regardless of stimulated modality. Posterior superior temporal sulcus showed sustained delay activity, irrespective of sensory modality, side, and response type. We conclude that the delay activations reflect translation of sensory signals into effector-specific motor circuits in parietal and frontal cortex (plus an impact on contralateral visual cortex for planned saccades), regardless of cue modality; while posterior STS provides a representation that generalises across both sensory modality and motor effector.