Visual acuity is greatest in the centre of the visual field, peaking in the fovea and degrading significantly towards the periphery. The rate of decay of visual performance with eccentricity depends strongly on the stimuli and task used in measurement. While detailed measures of this decay have been made across a broad range of tasks, a comprehensive theoretical account of this phenomenon is lacking. We demonstrate that the decay in visual performance can be attributed to the efficient encoding of binocular information in natural scenes. The efficient coding hypothesis holds that the early stages of visual processing attempt to form an efficient coding of ecologically valid stimuli. Using Independent Component Analysis to learn an efficient coding of stereoscopic images, we show that the ratio of binocular to monocular components varied with eccentricity at the same rate as human stereo acuity and Vernier acuity. Our results demonstrate that the organisation of the visual cortex is dependent on the underlying statistics of binocular scenes and, strikingly, that monocular acuity depends on the mechanisms by which the visual cortex processes binocular information. This result has important theoretical implications for understanding the encoding of visual information in the brain.