Objective: To investigate the relations between premorbid and current mental ability, mood, and white matter signal abnormalities detected by T2 weighted brain magnetic resonance imaging (MRI) and impairment of balance and mobility in older adults. Methods: 97 subjects from the Aberdeen 1921 birth cohort underwent brain MRI, evaluation of balance, and measurement of gait speed. White matter hyperintensities detected on T2 weighted MRI scans were rated by three independent raters on three variables: white matter lesions; periventricular lesions; and brain stem lesions. Results: Decreased gait speed was correlated with impaired visual acuity (p = 0.020), shorter stature (p = 0.008), a lower childhood IQ (p = 0.030), a lower current Raven’s progressive matrices score (Raven score) (p < 0.001), a higher hospital anxiety and depression scale (HADS) score (p = 0.004), and an increased grade of brain stem lesions on MRI. Inability to balance was correlated with Raven score (p = 0.042), brain stem lesions (p = 0.003), white matter lesions (p = 0.003), and periventricular lesions (p = 0.038). Binary logistic regression identified brain stem lesions (odds ratio (OR) 0.22; 95% confidence interval 0.09 to 0.54) and HADS depression score (OR 0.75; 0.58 to 0.97) as the only significant associations with balance. Structural equation modelling detected an association between two latent traits representing white matter disease and an integrating function, respectively. Conclusions: In this cohort, white matter lesions, periventricular lesions, and brain stem lesions were associated with impaired balance. Current mental ability was strongly related to gait speed. There appears to be a concordance between motor skills and intellect in old age, which is degraded by white matter disease.