Extended Reality (XR) is a powerful tool for training, education, and gaming. Research suggests that gender differences exist in XR environments including women having a lower sense of subjective presence and being more susceptible to motion sickness. However, the underrepresentation of women both as participants and researchers could lead to potential design biases, impacting the accuracy and inclusivity of XR systems. This work investigates subtle design differences in virtual environments on women's performance on a cognitive test. Non-male participants (n = 40) completed the Stroop Interference Task in two virtual classroom environments: a neutral and a stereotypically STEM environment. The environments were altered by four wall posters depicting positive gender-neutral and nature posters to science-fiction and positive male figures, such as Albert Einstein. Results support that when participants were in the stereotypical environment they were more distracted and responded more slowly and less accurately than when they were in the neutral environment. Additionally, positive female self-avatars buffered participants from the negative impacts of the stereotypical environment. These results highlight the need for more inclusive research practices. Minor adjustments can significantly improve or harm women's engagement and performance in XR settings. We emphasize the importance of bias awareness in study design, and recommend that researchers consider how their experiments could impact participants of all demographics, in order to enhance inclusivity and non-biased results.