The smooth pursuit eye movement (SPEM) system is much more sensitive to target motion perturbations during pursuit than during fixation. This sensitivity is commonly attributed to a dynamic gain control mechanism. Neither the neural substrate nor the functional architecture for this gain control has been fully revealed. There are at least two cortical areas that crucially contribute to smooth pursuit and are therefore eligible sites for dynamic gain control: The medial superior temporal area (MST) and the pursuit area of the frontal eye fields (FEF), which both project to brainstem premotor structures via parallel pathways. The aim of this study was to develop a model of smooth pursuit based on behavioral, anatomical, and neurophysiological results to account for nonlinear dynamic gain control. Using a behavioral paradigm in humans consisting of a sinusoidal oscillation (4 Hz, ±8 deg/s) superimposed on a constant velocity target motion (0-24 deg/s), we were able to identify relevant gain control parameters in the model. A salient feature of our model is the emergence of two parallel pathways from higher visual cortical to lower motor areas in the brainstem that correspond to the MST and FEF pathways. Detailed analysis of the model revealed that one pathway mainly carries eye velocity related signals, while the other is mostly associated with eye acceleration. From comparison with known neurophysiological results we conclude that the dynamic gain control can be attributed to the FEF pathway, while the MST pathway serves as the basic circuit for maintaining an ongoing SPEM.