Eye contact is a powerful social stimulus for human and non-human primates. However, it is unclear whether brain mechanisms that interpret eye contact are sensitive to gender. Here we show that human brain responses to eye contact are indeed gender specific. Recording event-related potentials directly from the medial temporal lobes, we found that eye contact elicited specific responses in men only when they saw female faces. Conversely, women responded specifically to eye contact only when they saw pictures of men. Thus, the human medial temporal lobes subserve specifically the processing of eye contact with persons of the opposite gender.