Alzheimer’s disease (AD) is characterized by substantial neurodegeneration, including both cortical atrophy and loss of underlying white matter fiber tracts. Understanding longitudinal alterations to white matter may provide new insights into trajectories of brain change in both healthy aging and AD, and fluid biomarkers may be particularly useful in this effort. To examine this, 151 late-middle-aged participants enriched with risk for AD with at least one lumbar puncture and two diffusion tensor imaging (DTI) scans were selected for analysis from two large observational and longitudinally followed cohorts. Cerebrospinal fluid (CSF) was assayed for biomarkers of AD-specific pathology (phosphorylated-tau/Aβ42 ratio), axonal degeneration (neurofilament light chain protein, NFL), dendritic degeneration (neurogranin), and inflammation (chitinase-3-like protein 1, YKL-40). Linear mixed effects models were performed to test the hypothesis that biomarkers for AD, neurodegeneration, and inflammation, or two-year change in those biomarkers, would be associated with worse white matter health overall and/or progressively worsening white matter health over time. At baseline in the cingulum, phosphorylated-tau/Aβ42 was associated with higher mean diffusivity (MD) overall (intercept) and YKL-40 was associated with increases in MD over time. Two-year change in neurogranin was associated with higher mean diffusivity and lower fractional anisotropy overall (intercepts) across white matter in the entire brain and in the cingulum. These findings suggest that biomarkers for AD, neurodegeneration, and inflammation are potentially important indicators of declining white matter health in a cognitively healthy, late-middle-aged cohort.