Dyslipidemia is common in adults and contributes to high rates of cardiovascular disease and may be linked to subsequent neurodegenerative and neurovascular diseases. This study examined whether lower brain volumes and cognition associated with dyslipidemia could be observed in cognitively healthy adults, and whether apolipoprotein E (APOE) genotype or family history of Alzheimer’s disease (FHAD) alters this effect. T1-weighted magnetic resonance imaging was used to examine regional brain gray matter (GM) and white matter (WM) in 183 individuals (58.4 +/- 8.0 years) using voxel-based morphometry. A non-parametric multiple linear regression model was used to assess the effect of high-density lipoprotein (HDL) and non-HDL cholesterol, APOE, and FHAD on regional GM and WM volume. A post hoc analysis was used to assess whether any significant correlations found within the volumetric analysis had an effect on cognition. HDL was positively correlated with GM volume in the bilateral temporal poles, middle temporal gyri, temporo-occipital gyri, and left superior temporal gyrus and parahippocampal region. This effect was independent of APOE and FHAD. A significant association between HDL and the Brief Visuospatial Memory Test was found. Additionally, GM volume within the right middle temporal gyrus, the region most affected by HDL, was significantly associated with the Controlled Oral Word Association Test and the Center for Epidemiological Studies Depression Scale. These findings suggest that adults with decreased levels of HDL cholesterol may be experiencing cognitive changes and GM reductions in regions associated with neurodegenerative disease and therefore, may be at greater risk for future cognitive decline.