The study examined the effect of age on subjects’ performance on phonological and visual working memory tasks. The aim was to determine whether the well-documented age-related decrease in working memory ability is general (whereby there is an equivalent deficit across both components) or selective (whereby one component declines more quickly than the other) by comparing the modality of encoding used in immediate serial recall. A young adult group (20-25 years) and an old adult group (60-79 years) were presented with 4-letter sequences for immediate serial recall. There was an aural condition and a visual condition. In the aural condition, half of the sequences comprised acoustically similar letters (C, B, D, P, T, V), whereas the other half comprised acoustically distinct letters (F, H, L, Q, R, Y). In the visual condition, half of the sequences included letters whose upper-case and lower-case forms are visually similar (Cc, Kk, Pp, Ss, Vv, Ww), whereas the other half included letters whose upper-case and lower-case forms are visually distinct (Bb, Dd, Gg, Ll, Qq, Rr). The young adult group performed better than the old adult group in all 4 conditions. Both the young and the old adult groups demonstrated a phonological similarity effect, whereby performance was better on acoustically distinct sequences than acoustically similar sequences. Whilst the young adult group also demonstrated a visual similarity effect, the old adult group performed poorly on both visually similar and visually distinct sequences, demonstrating only a slight effect of similarity. The findings demonstrate that old adults can rely less upon visual working memory than young adults, and that there is a selective age-related deficit, with visual working memory being more age-sensitive than phonological working memory.