Cognitive decline is one of the hallmark features of Alzheimer’s disease, but many studies struggle to find strong associations between cognitive function and genetic variants. In order to identify which aspects of cognition are more likely to have a strong genetic component, we assessed the heritability of various cognitive functions related to Alzheimer’s disease in 303 initially asymptomatic middle-aged adult siblings with a parental history of Alzheimer’s disease from the Wisconsin Registry for Alzheimer’s Prevention. Participants underwent extensive cognitive testing, and six cognitive factors were identified via factor analysis. Working Memory and Visual Learning & Memory had the highest heritability (52% and 41%, respectively). Inclusion of APOE allele counts did not notably change heritability estimates, indicating that there are likely additional genetic variants contributing to cognition. These findings suggest that future genetic studies should focus on the cognitive domains of Working Memory and Visual Learning & Memory.