Can Pulling the Lower Eye Lid Cause Sagging?

The eyelids do indeed have thin and relatively delicate skin compared to other areas of the body, but the eyelid also includes muscle, fat, and cartilage, all of which comprise a surprisingly durable and stretchy structure. Otherwise, those of us who suffer from allergies and rub our eyes every morning and throughout the day would have the saggiest eyelids ever! This, obviously, does not occur!

Aging and innate laxity from genetic causes are what make us candidates for cosmetic eyelid surgery, not stretching the lids for several seconds. You need not be concerned, unless your eyelids had poor tone to begin with, and this just drew your attention to what already existed! If so, rest assured your doctor checking your eye infection had nothing whatsoever to do with your present situation, except to appropriately treat your infection.

Just the inflammation from the eye infection could weaken the eyelid.  A single examination is unlikely to cause long term eyelid sagging.  However, it does not sound like there is much option here.  Treatment requires detailed examination.

The circumstances you describe should not lead to any permanent damage to your lower eyelid skin. It is true that the lower eyelid skin is very delicate, and as we age the eyelid can develop some laxity that could be aggravated be constantly pulling and dragging on the lower eyelid skin. Other factors such as lower eyelid surgery, sun damage, trauma, etc. could also contribute to situations causing lower eyelid laxity, also commonly called "ectropion"; if you have any concerns, you can check with a plastic surgeon or eye professional