This study investigated differences in retrospective cognitive trajectories between amyloid and tau PET biomarker stratified groups in initially cognitively unimpaired participants sampled from the Wisconsin Registry for Alzheimer’s Prevention. One hundred and sixty-seven initially unimpaired individuals (baseline age 59 ± 6 years; 115 females) were stratified by elevated amyloid-β and tau status based on 11C-Pittsburgh compound B (PiB) and 18F-MK-6240 PET imaging. Mixed effects models were used to determine if longitudinal cognitive trajectories based on a composite of cognitive tests including memory and executive function differed between biomarker groups. Secondary analyses investigated group differences for a variety of cross-sectional health and cognitive tests, and associations between 18F-MK-6240, 11C-PiB, and age. A significant group × age interaction was observed with post hoc comparisons indicating that the group with both elevated amyloid and tau pathophysiology were declining approximately three times faster in retrospective cognition compared to those with just one or no elevated biomarkers. This result was robust against various thresholds and medial temporal lobe regions defining elevated tau. Participants were relatively healthy and mostly did not differ between biomarker groups in health factors at the beginning or end of study, or most cognitive measures at study entry. Analyses investigating association between age, MK-6240 and PiB indicated weak associations between age and 18F-MK-6240 in tangle-associated regions, which were negligible after adjusting for 11C-PiB. Strong associations, particularly in entorhinal cortex, hippocampus and amygdala, were observed between 18F-MK-6240 and global 11C-PiB in regions associated with Braak neurofibrillary tangle stages I-VI. These results suggest that the combination of pathological amyloid and tau is detrimental to cognitive decline in preclinical Alzheimer’s disease during late middle-age. Within the Alzheimer’s disease continuum, middle-age health factors likely do not greatly influence preclinical cognitive decline. Future studies in a larger preclinical sample are needed to determine if and to what extent individual contributions of amyloid and tau affect cognitive decline. 18F-MK-6240 shows promise as a sensitive biomarker for detecting neurofibrillary tangles in preclinical Alzheimer’s disease.