In optical tweezers, thermal drift is detrimental for high-resolution measurements. In particular, absorption of the trapping laser light by the microscope objective that focuses the beam leads to heating of the objective and subsequent drift. This entails long equilibration times which may limit sensitive biophysical assays. Here, we introduce an objective temperature feedback system for minimizing thermal drift. We measured that the infrared laser heated the objective by 0.7 K per watt of laser power and that the laser focus moved relative to the sample by approximately 1 nm/mK due to thermal expansion of the objective. The feedback stabilized the temperature of the trapping objective with millikelvin precision. This enhanced the long-term temperature stability and significantly reduced the settling time of the instrument to about 100 s after a temperature disturbance while preserving single DNA base-pair resolution of surface-coupled assays. Minimizing systematic temperature changes of the objective and concurrent drift is of interest for other high-resolution microscopy techniques. Furthermore, temperature control is often a desirable parameter in biophysical experiments.