Primates have a remarkable ability to perceive, recognize, and discriminate among the plethora of people, places, and things that they see, and neural selectivity in the primate inferotemporal (IT) cortex is thought to underlie this ability. Here we investigated the relationship between neural response and perception by recording from IT neurons in monkeys while they viewed realistic images. We then compared the similarity of neural responses elicited by images to the quantitative similarity of the images. Image similarity was approximated using several algorithms, two of which were designed to search image databases for perceptually similar images. Some algorithms for image similarity correlated well with human perception, and these algorithms explained part of the stimulus selectivity of IT neurons. Images that elicited similar neural responses were ranked as more similar by these algorithms than images that elicited different neural responses, and images ranked as similar by the algorithms elicited similar responses from neurons. Neural selectivity was predicted more accurately when the reference images for algorithm similarity elicited either very strong or very weak responses from the neuron. The degree to which algorithms for image similarity were correlated with human perception was related to the degree to which algorithms explained the selectivity of IT neurons, providing support for the proposal that the selectivity of IT neurons is related to perceptual similarity of images.