The Trail Making Test part B (TMT-B) is highly sensitive to age-related changes in the brain and cognitive function. However, the precise contribution of periventricular hyperintensities (PVH), deep white matter hyperintensities (DWMH), and medial temporal lobe atrophy (MTA) to task performance remains unspecified. Similarly, diminished performance may be due to deficient flexibility functions, but also to other age-related cognitive decline (e.g., mental slowing). The aim of the present study was to determine neuroanatomical (PVH, DWMH, MTA) and neuropsychological (working memory, executive function, speed and attention, episodic memory) predictors of TMT-B performance in elderly people. Results showed that MTA was the strongest predictor of TMT-B performance. The predictive value of the neuropsychological scores differed among the various TMT-B variables. For example, all neuropsychological domains predicted the TMT-B total completion time, whereas only executive function predicted the ratio score (TMT-B/A). We conclude that MTA is a very important predictor of TMT-B performance in elderly people. Furthermore, multiple cognitive functions are involved in TMT-B performance and a mild decline in any of these functions may result in diminished TMT-B performance. Therefore it is crucial to use the ratio score when one wishes to examine executive function ability.