Cardiorespiratory fitness (CRF) is an objective measure of habitual physical activity (PA), and has been linked to increased brain structure and cognition. The gold standard method for measuring CRF is graded exercise testing (GXT), but GXT is not feasible in many settings. The objective of this study was to examine whether a non-exercise estimate of CRF is related to gray matter (GM) volumes, white matter hyperintensities (WMH), cognition, objective and subjective memory function, and mood in a middle-aged cohort at risk for Alzheimer’s disease (AD). Three hundred and fifteen cognitively healthy adults (mean age =58.58 years) enrolled in the Wisconsin Registry for Alzheimer’s Prevention underwent structural MRI scanning, cognitive testing, anthropometric assessment, venipuncture for laboratory tests, and completed a self-reported PA questionnaire. A subset (n = 85) underwent maximal GXT. CRF was estimated using a previously validated equation incorporating sex, age, body-mass index, resting heart rate, and self-reported PA. Results indicated that the CRF estimate was significantly associated with GXT-derived peak oxygen consumption, validating its use as a non-exercise CRF measure in our sample. Support for this finding was seen in significant associations between the CRF estimate and several cardiovascular risk factors. Higher CRF was associated with greater GM volumes in several AD-relevant brain regions including the hippocampus, amygdala, precuneus, supramarginal gyrus, and rostral middle frontal gyrus. Increased CRF was also associated with lower WMH and better cognitive performance in Verbal Learning & Memory, Speed & Flexibility, and Visuospatial Ability. Lastly, CRF was negatively correlated with self- and informant-reported memory complaints, and depressive symptoms. Together, these findings suggest that habitual participation in physical activity may provide protection for brain structure and cognitive function, thereby decreasing future risk for AD.