To mitigate capacity limits of working memory, people allocate resources according to an item's relevance. However, the neural mechanisms supporting such a critical operation remain unknown. Here, we developed computational neuroimaging methods to decode and demix neural responses associated with multiple items in working memory with different priorities. In striate and extrastriate cortex, the gain of neural responses tracked the priority of memoranda. We decoded higher-priority memoranda with smaller error and lower uncertainty. Moreover, these neural differences predicted behavioral differences in memory prioritization between and within participants. Trial-wise variability in the magnitude of delay activity in the frontal cortex predicted differences in decoded precision between low- and high-priority items in visual cortex. These results support a model in which feedback signals broadcast from frontal cortex sculpt the gain of memory representations in the visual cortex according to behavioral relevance, thus identifying a neural mechanism for resource allocation.