We previously identified sex-specific genetic loci associated with memory performance, a strong Alzheimer's disease (AD) endophenotype. Here, we expand on this work by conducting sex-specific, cross-ancestral, genome-wide meta-analyses of three cognitive domains (memory, executive functioning, and language) in 33,918 older adults (57% female; 41% cognitively impaired; mean age=73 years) from 10 aging and AD cohorts. All three domains were comparably heritable across sexes. Genome-wide meta-analyses identified three novel loci: a female-specific language decline-associated locus, VRK2 (rs13387871), which is a published candidate for neuropsychiatric traits involving language ability; a male-specific memory decline-associated locus among cognitive impaired, DCHS2 (rs12501200), which is a published candidate gene for AD age-at-onset; and a sex-interaction with baseline executive functioning, AGA (rs1380012), among cognitive impaired. We additionally provide evidence for shared genetic architecture between lifetime estrogen exposure and AD-related cognitive decline. Overall, we identified sex-specific variants, genes, and pathways relating to three cognitive domains among older adults.