In general, leave your aligners in when you drink and most things are okay...have yet to see a cavity along the edge of an aligner even with poor hygiene. With fruit infused water, watch out for excessive acidity which might make you prone to excess enamel wear during toothbrushing (abrasion)...
Since the greater the time you are wearing your aligners, the shorter the length of your treatment. Ii strongly advise that you replace your aligners, after of course, either brushing or swallowing, asap.
I have had other patients inquire about this and I am also not a fan of plastics in general. However, in the years of using Invisalign as a safe and effective treatment with my patients, I have never had an adverse reaction to the material. It is factory-produced long chain polymer material...