Is it ok to workout and play sports like soccer and basketball two weeks after surgery?

I am trying to get a better idea of exercising after gynecomastia surgery. I played soccer exactly 2 weeks after surgery, which is the day I had my second follow up appointment with my surgeon and he allowed me to play sports but he said to avoid any chest exercises. However, when I played, I got hit hard in my chest which didn't cause any pain but I'm wondering how any hit or contact affects my chest, especially that hit? Can I play sports that early or should I avoid it to be on the safe side?