Can a breast expander's port get damaged?

Usually not but would check with your surgeon to see in person... very rarely is there a mechanical issue with the tissue expander

The expanders I use have the port integrated into the expander, so there are no tubes or remote injecting sites. It sound like you may have one of the models that do have a remote port.
You shouldn't "feel" any fluid around the port. You may be feeling fluid within a partially filled tissue expander.
Either way, I think you should go back and see you plastic surgeon to make sure that all is well.

Typically the port is fine.  there be a small amount of residual fluid around it from the initial surgery.  The surgeon can examine it to give you a better idea.