Breast Tissue in UOQ of One Breast "Disintegrating" Causing a Sloping Appearance, What Could Be Causing This?

Years ago I was having pain in the upper outer quadrant of my left breast. The breast tissue in this area continues to "disintegrate" for lack of a better word. There is not much breast tissue there, it slopes down and causes the nipple to flip up. My doctor said the coopers ligaments were weakening and to wear a supportive bra. However, I am concerned because it continues to get get worse and I am only 28 years old. Also, there is pain/soreness in this area. Could it be something more serious?