Human aging is accompanied by progressive changes in executive function and memory, but the biological mechanisms underlying these phenomena are not fully understood. Using neurite orientation dispersion and density imaging, we sought to examine the relationship between age, cellular microstructure, and neuropsychological scores in 116 late middle-aged, cognitively asymptomatic participants. Results revealed widespread increases in the volume fraction of isotropic diffusion and localized decreases in neurite density in frontal white matter regions with increasing age. In addition, several of these microstructural alterations were associated with poorer performance on tests of memory and executive function. These results suggest that neurite orientation dispersion and density imaging is capable of measuring age-related brain changes and the neural correlates of poorer performance on tests of cognitive functioning, largely in accordance with published histological findings and brain-imaging studies of people of this age range. Ultimately, this study sheds light on the processes underlying normal brain development in adulthood, knowledge that is critical for differentiating healthy aging from changes associated with dementia.