Virus particle formation of HIV-1 is driven by the viral structural protein Gag. In most cell types including T cells, Gag assembles into virus particles at the plasma membrane whereas, in HIV-1-infected macrophages, Gag and virus particles have been shown to accumulate in intracellular vesicles. At the moment, what causes this difference between cell types remains unknown. However, recent findings on the relationships between Gag and the cellular membrane system have substantially increased our understanding of the mechanisms by which sites of virus assembly are determined. I will review our current knowledge regarding the roles played by endosomal trafficking pathways, membrane microdomains, and plasma membrane lipids, and discuss the physiological significance of the interactions between Gag and specific membrane structures.