There's really no hard and fast answer to this. In general, the more you wear contacts prior to your measurements for surgery, the higher the chance the cornea will not be back to its "natural" shape, and the less accurate the surgical result. The less time elapsed between contact lens wear and measurements, and the longer the time of wear, the worse the situation. A "few hours" five days before measurements may not make a huge difference, but it could make a slight difference.
It also depends on what type of lens is being worn--a gas permeable lens will potentially cause more corneal changes than a soft lens.
If it's not terribly inconvenient, the ideal solution would be to follow your doctor's directions and truly stay out of lenses the prescribed period of time before your surgical measurements are taken. If you don't mind having a slightly increased risk of needing a touchup or enhancement, then I wouldn't worry about it. There is no data I'm aware of which quantifies all this.
I will say in my practice, I've seen nearly everything, including patients who refuse to take their lenses out the prescribed period of time, and are willing to accept a potentially less accurate correction. As long as the patient is fully informed, and the corneal shape is within the bounds of normal (no inferior corneal steepening, or keratoconus), I will proceed if they wish.