Both major depressive disorder and bipolar disorder are the subject of a voluminous imaging and genetics literature. Here, we attempt a comprehensive review of MRI and metabolic PET studies conducted to date on these two disorders, and interpret our findings from the perspective of developmental and degenerative models of illness. Elevated activity and volume loss of the hippocampus, orbital and ventral prefrontal cortex are recurrent themes in the literature. In contrast, dorsal aspects of the PFC tend to display hypometabolism. Ventriculomegaly and white matter hyperintensities are intimately associated with depression in elderly populations and likely have a vascular origin. Important confounding influences are medication, phenotypic and genetic heterogeneity, and technological limitations. We suggest that environmental stress and genetic risk variants interact with each other in a complex manner to alter neural circuitry and precipitate illness. Imaging genetic approaches hold out promise for advancing our understanding of affective illness.