Had deviated septum surgery 4 days ago. When something comes in contact with my nose will it affect it? Do I have to worry?

Since you are only 4 days out you should be taking it easy and not over exerting yourself. If you over exert yourself you run the risk of bleeding, prolonged swelling, shifting of the nasal septum, scarring and poor healing. This is a surgery that you only want done once if possible due to the downtime. I will have patients exercising normally after 2 weeks and playing all sports in 3 weeks after septoplasty but you should ask your surgeon as it will vary depending on how much was done to the nose. 

You should follow the postoperative instructions of your surgeon. Contact sports are probably not a great idea in the early postoperative period. Because there are many different maneuvers for septal surgery, it is best to check with your surgeon.