I am considering having my implants removed. If I do not have them replaced, what if any downside is there?

You can elect to have your implants removed at any time.  Based on the information given you will likely have loose skin  when the implants are removed.  If desired you may choose to have a lift to improve shape.  I encourage you to meet with a board certified plastic surgeon near you to learn more.

After explantation alone, many women are dissatisfied because of the hanging skin, empty appearance and often require a lift.  I recommend explantation and a lift simultaneously using the no vertical scar technique.  The implants are removed, your breast tissue is reshaped creating upper pole fullness, elevated higher on the chest wall and more medial to increase your cleavage.  Vertical scars are avoided and nipple sensation (in 95%) is maintained, if not injured during your previous surgery.  Later, fat transfers can be performed if additional volume is desired.  Implants are not lifetime devices.  8% fail within the first year and 30% fail within six years.  Fat transfers are permanent and incorporated into the body.Best Wishes,Gary Horndeski, M.D.

The downsides of removal are primarily aesthetic.It is difficult to make 100% accurate predictions following explant. There are a number of factors to consider including age (the breast changes over time irregardless of augmentation), weight change, skin stretch and residual elasticity.
Following removal, one can expect:
-Loss of volume- Not an exact correlation between pre- and post augmentation due to the changes over time. You may be bigger or smaller than you remember.
-Ptosis/sagging-Ptosis develops over time and may be worsened by the presence of an implant. If a patient has pre-explant sagging, removal is likely to worsen this to a certain extent. If there is no ptosis, it does not necessarily mean there will be no ptosis after removal.
-Skin laxity/stretch- There is inherent skin elasticity which will allow for some measure of snap back. However, the greater the volume differential pre- and post- the more likely there will be excess skin.
In my experience, it is usually a safe assumption that there will be some changes post removal which oftentimes may necessitate further operative intervention (i.e. lift), however this is often a function of size and length of time post op. Small implants which have only been in place for a short period will lead to an improved response/recoil post removal. Whereas larger implants which have been in for several years will often necessitate a lift.
With regards to your specific question, the recall does not require explantation in asymptomatic patients. If you are not having issues, you do not have to have the devices removed.
As always, discuss your concerns with a board certified plastic surgeon (ABPS).
Donovan Rosas MD
Board Certified by the American Board of Plastic Surgery
Member: American Society for Aesthetic Plastic Surgery, American Society of Plastic Surgeons
RealSelf Verified Member
RealSelf Top 100
RealSelf Hall of Fame

Dear Neat8522,
If you don't have any symptoms with your breasts, FDA does not recommend removal of your implants. However, if you want to remove them, you can do it. There are some options for patients to try to restore some of the fullness, like fat transfer or breast lift. 
Daniel Barrett, MD
Certified, American Board of Plastic Surgery
Member, American Society of Plastic Surgery
Member, American Society of Aesthetic Plastic Surgery

The FDA has not recommended removal or replacement of textured breast implants or tissue expanders in asymptomatic patients.  Textured implants could be more difficult to remove because of the friction caused between the implant and the muscle surface.  Leaving a capsule behind is preferred if it is thin and pliable. If there are calcific components, or silicone granulomatous material, or if the capsule is deforming the breast tissue, then removal of the capsule may be warranted.  If the capsule is left behind, the body will break down and remodel the scar tissue on its own.The smaller the implant and the less the amount of time it has been since placement, the greater the chance the breasts will return to their preoperative state. It is very difficult to predict breast size following explant b/c breast implants and weight fluctuations over time tend to thin tissues, cause a decrease in size, and create more droop.  
It is not unreasonable to remove the implants and then observe a waiting period of six months before the determination for breast lift is made. This will give your breasts time to bounce back and your PS time to determine what type of lift, if any, is needed. 
Consult in person with a board certified PS who can examine you, order tests if necessary, and determine if you need to remove the implants and which implant removal procedure would be best