Visual perception and memory are the most important components of vision processing in the brain. It was thought that the perceptual aspect of a visual stimulus occurs in visual cortical areas and that this serves as the substrate for the formation of visual memory in a distinct part of the brain called the medial temporal lobe. However, current evidence indicates that there is no functional separation of areas. Entire visual cortical pathways and connecting medial temporal lobe are important for both perception and visual memory. Though some aspects of this view are debated, evidence from both sides will be explored here. In this review, we will discuss the anatomical and functional architecture of the entire system and the implications of these structures in visual perception and memory.