Surgery and anaesthesia might affect cognition in middle-aged people without existing cognitive dysfunction. We measured memory and executive function in 964 participants, mean age 54 years, and again four years later, by when 312 participants had had surgery and 652 participants had not. Surgery between tests was associated with a decline in immediate memory by one point (out of a maximum of 30), p = 0.013: memory became abnormal in 77 out of 670 participants with initially normal memory, 21 out of 114 (18%) of whom had had surgery compared with 56 out of 556 (10%) of those who had not, p = 0.02. The number of operations was associated with a reduction in immediate memory on retesting, beta coefficient (SE) 0.08 (0.03), p = 0.012. Working memory decline was also associated with longer cumulative operations, beta coefficient (SE) -0.01 (0.00), p = 0.028. A reduction in cognitive speed and flexibility was associated with worse ASA physical status, beta coefficient (SE) 0.55 (0.22) and 0.37 (0.17) for ASA 1 and 2 vs. 3, p = 0.035. However, a decline in working memory was associated with better ASA physical status, beta coefficient (SE) -0.48 (0.21) for ASA 1 vs. 3, p = 0.01.