I am considering having my implants removed. If I do not have them replaced, what if any downside is there?

Dear Neat8522,
If you don't have any symptoms with your breasts, FDA does not recommend removal of your implants. However, if you want to remove them, you can do it. There are some options for patients to try to restore some of the fullness, like fat transfer or breast lift. 
Daniel Barrett, MD
Certified, American Board of Plastic Surgery
Member, American Society of Plastic Surgery
Member, American Society of Aesthetic Plastic Surgery

Thank you for the question.  You are not alone: many women considering breast implant removal have the same concerns when it comes to what their breasts will look like without breast implants.  There are maneuvers such as breast lifting that may be helpful at the time of breast implant removal surgery.  
 
Generally speaking, what your breasts will look like after explantation depends on several factors such as: the quality of skin elasticity (the better the elasticity the better the skin will bounce back), the size of the implants used (the larger the implant the more trouble you may have with redundant skin), and the amount of breast tissue present at this time (which may have changed since the time of your breast augmentation).
Life experience since your breast augmentation procedure, such as pregnancy or weight gain weight loss, will potentially influence the factors discussed above. If you take these factors into consideration and apply them to your specific circumstances you may get a good idea of what to expect after the implants are removed.
 Patience is definitely indicated after removal of breast implants; it will take many months/year for the skin to “bounce back” and for you to see the final appearance of the breasts.
In "borderline" cases where there is doubt whether breast lifting is necessary or not, I advise patients to undergo explantation only and to give their breasts the benefit of the doubt  before deciding whether breast lifting would be beneficial.  
I hope this helps. Best wishes.