Breast Shrunk w/ Exercise. Will I Get Back the Fat I Lost?

hi im slim and 17 years, about 100 lbs..i used to do karate and my breasts were firm and seemed bigger. when i stopped, they did not seem as full and drooped a bit. so i started to do really intense exercise on my breasts again. i pushed against the wall really hard to work my upper body...and at the same time i did swimming. after, i realized my breasts shrunk, got smaller. will i ever get back the fat i lost or will they continue to grow as is??