In the cerebral cortex, neuronal circuits are first laid down by intrinsic mechanisms and then refined by experience. In the canonical model, this refinement is driven by activity-dependent competition between inputs for some limited cortical resource. Here we examine this idea in the mouse visual cortex, at the peak of the critical period for experience-dependent plasticity. By imaging intrinsic optical responses, we mapped the strength and size of each eye's cortical representation in normal mice, mice that had been deprived of patterned vision unilaterally or bilaterally, and in mice in which the contralateral eye had been removed. We find that for both eyes, a period of visual deprivation results in a loss of cortical responsiveness to stimulation through the deprived eye. In addition, the ipsilateral eye pathway is affected by the quality of vision through the opposite eye. Our findings indicate that while both contralateral and ipsilateral eye pathways require visual experience for their maintenance, ipsilateral eye projections bear an additional, unique sensitivity to binocular interactions.