Alzheimer's disease (AD), the most common form of dementia, is marked by cognitive decline and amyloid-beta (Aβ) plaque deposition in the brain. Microglia cluster around Aβ plaques shifting to a plaque-associated microglia (PAM) immunophenotype. This study investigates the association between Aβ, microglia and PAM with cognitive performance in 75 older adults, 39 with normal cognition and 36 with AD. Postmortem brain samples containing the middle temporal gyrus (MTG) underwent duplex immunohistochemistry for ionized calcium-binding adaptor molecule 1 (IBA1) and Aβ. A machine learning pipeline quantified parameters of Aβ, microglia and PAM expression. This study evaluated sex- and layer-specific patterns of expression of these parameters and the relationship with global cognitive performance, as measured by the Cognitive Abilities Screening Instrument (CASI) and the Mini-Mental State Examination (MMSE). Additionally, four specific cognitive domains, memory, executive function, language, and visuospatial processing, were evaluated. Aβ and PAM were significantly higher in AD, with no sex- or layer-specific differences. In layers 3 and 4, Aβ plaque size was inversely correlated with MMSE. In all layers, total and activated microglial densities were related to executive function, but in a cognitive status-dependent manner. In layers 5 and 6, higher PAM expression correlated with lower CASI and MMSE scores. PAM expressions in layers 3, 5, and 6 were negatively associated with memory scores. This study characterizes Aβ, microglia, and PAM patterns in the MTG, revealing layer-specific associations between histopathological metrics and cognitive domains. It also highlights PAM as a potential therapeutic target to mitigate cognitive decline in AD.