I am considering having my implants removed. If I do not have them replaced, what if any downside is there?

The downsides of removal are primarily aesthetic.It is difficult to make 100% accurate predictions following explant. There are a number of factors to consider including age (the breast changes over time irregardless of augmentation), weight change, skin stretch and residual elasticity.
Following removal, one can expect:
-Loss of volume- Not an exact correlation between pre- and post augmentation due to the changes over time. You may be bigger or smaller than you remember.
-Ptosis/sagging-Ptosis develops over time and may be worsened by the presence of an implant. If a patient has pre-explant sagging, removal is likely to worsen this to a certain extent. If there is no ptosis, it does not necessarily mean there will be no ptosis after removal.
-Skin laxity/stretch- There is inherent skin elasticity which will allow for some measure of snap back. However, the greater the volume differential pre- and post- the more likely there will be excess skin.
In my experience, it is usually a safe assumption that there will be some changes post removal which oftentimes may necessitate further operative intervention (i.e. lift), however this is often a function of size and length of time post op. Small implants which have only been in place for a short period will lead to an improved response/recoil post removal. Whereas larger implants which have been in for several years will often necessitate a lift.
With regards to your specific question, the recall does not require explantation in asymptomatic patients. If you are not having issues, you do not have to have the devices removed.
As always, discuss your concerns with a board certified plastic surgeon (ABPS).
Donovan Rosas MD
Board Certified by the American Board of Plastic Surgery
Member: American Society for Aesthetic Plastic Surgery, American Society of Plastic Surgeons
RealSelf Verified Member
RealSelf Top 100
RealSelf Hall of Fame

Dear Neat8522,
If you don't have any symptoms with your breasts, FDA does not recommend removal of your implants. However, if you want to remove them, you can do it. There are some options for patients to try to restore some of the fullness, like fat transfer or breast lift. 
Daniel Barrett, MD
Certified, American Board of Plastic Surgery
Member, American Society of Plastic Surgery
Member, American Society of Aesthetic Plastic Surgery