Endophenotypes (intermediate phenotypes) are objective, heritable, quantitative traits hypothesized to represent genetic risk for polygenic disorders at more biologically tractable levels than distal behavioural and clinical phenotypes. It is theorized that endophenotype models of disease will help to clarify both diagnostic classification and aetiological understanding of complex brain disorders such as obsessive-compulsive disorder (OCD). To investigate endophenotypes in OCD, we measured brain structure using magnetic resonance imaging (MRI), and behavioural performance on a response inhibition task (Stop-Signal) in 31 OCD patients, 31 of their unaffected first-degree relatives, and 31 unrelated matched controls. Both patients and relatives had delayed response inhibition on the Stop-Signal task compared with healthy controls. We used a multivoxel analysis method (partial least squares) to identify large-scale brain systems in which anatomical variation was associated with variation in performance on the response inhibition task. Behavioural impairment on the Stop-Signal task, occurring predominantly in patients and relatives, was significantly associated with reduced grey matter in orbitofrontal and right inferior frontal regions and increased grey matter in cingulate, parietal and striatal regions. A novel permutation test indicated significant familial effects on variation of the MRI markers of inhibitory processing, supporting the candidacy of these brain structural systems as endophenotypes of OCD. In summary, structural variation in large-scale brain systems related to motor inhibitory control may mediate genetic risk for OCD, representing the first evidence for a neurocognitive endophenotype of OCD.