We explore theoretically and numerically the temporal contrast limitation of a self-referenced spectral interferometry measurement. An experimental confirmation is given by characterization and fine compression of hollow-core fiber generated sub-15 fs pulses, yielding an accurately measured coherent contrast of 50 dB on a ±400 fs time range.