It is unlikely that fever for a few days after your procedure would negatively impact your results. There are patients that have infections from the hair transplant procedure afterwards and they have fevers and their results are fine.
Fever itself may not affect the grafts or end result of a hair transplant surgery. However, fever may signify an infection or even graft rejection. In that case it may affect the end results of a hair transplant surgery. These are all "guesses". The best course of action is to follow up with your doctor for a recheck.
Transplanted hairs generally do very well with over 95% survival. Factors that affect survival include extraction technique, blood perfusion, and handling of the grafts.
Any shock or trauma after a transplant can affect the results. Especially if it is in the critical first 5-10 days. However, no researched has looked at this in detail so its hard to give a definitive answer. To reassure you, hair grafts tend to be very resilient in most people.
With your story, I would not worry. A flue with fever or such, will not impact a hair transplant at 4 days. No real risk unless you have an infection of the transplant