Honestly, no. At 4 days after surgery, that's when you will be at your most swollen and tender. Particularly with your physical job, I would recommend at least 7-10 days off, and no heavy lifting or upper body straining for 3 weeks.
Obviously, you can do whatever you want to do, however, in my practice, I tell patients not to resume strenuous activity for three weeks following surgery.