White matter hyperintensities (WMH) of presumed vascular origin, as seen on T2-weighted fluid attenuated inversion recovery magnetic resonance imaging, are known to increase with age and are elevated in Alzheimer’s disease (AD). The cognitive implications of these common markers are not well understood. Previous research has primarily focused on global measures of WMH burden and broad localizations that contain multiple white matter tracts. The aims of this study were to determine the pattern of WMH accumulation with age, risk for AD, and the relationship with cognitive function utilizing a voxel-wise analysis capable of identifying specific white matter regions. A total of 349 participants underwent T1-weighted and high-resolution T2-weighted fluid attenuated inversion recovery magnetic resonance imaging and neuropsychological testing. Increasing age and lower cognitive speed and flexibility (a component of executive function), were both significantly associated with regional WMH throughout the brain. When age was controlled, lower cognitive speed and flexibility was independently associated with WMH in the superior corona radiata. Apolipoprotein E ε4 and parental family history of AD were not associated with higher burden of WMH. The results contribute to a larger body of literature suggesting that white matter measures are linked with processing speed, and illustrate the utility of voxel-wise analysis in understanding the effect of lesion location on cognitive function.