Removing Lipoma on Vaginal Area?

A lump or mass in that area could be one of several possibilities, a Bartholin's cyst being one of them. At any rate, you indicated your doctor performed a rather brief exam, and your treatment cannot be undertaken based on that.
Any growing mass needs a full explanation, so please see a doc who can evaluate you fully.

Hello,
Soft tissue lumps are usually benign but not always. My tendency when I see such a mass, particularly if it is growing, is to recommend that it be removed. It is simpler when these lumps are small. Often, small ones can be removed under just local (numbing) in the doctor's office.
Who should remove it? That is debatable. I leave it to the patient's discretion. Plastic surgeons can often make the closure look better, but insurance coverage can be more dicey with plastic surgeons.