We show how to construct a linearly independent set of antisymmetrized geminal power (AGP) states, which allows us to rewrite our recently introduced geminal replacement models as linear combinations of non-orthogonal AGPs. This greatly simplifies the evaluation of matrix elements and permits us to introduce an AGP-based selective configuration interaction method, which can reach arbitrary excitation levels relative to a reference AGP, balancing accuracy and cost as we see fit.