Will my adult teeth fall out earlier to due using different retainers?

I had orthodontic treatment a few years ago and was given retainers. It cracked and wore a new retainer (less straight teeth). A year later I realised wore the old retainer and it pushed my teeth into a more straight position. However I recently lost that retainer and am now wearing the second retainer again (slightly cracked). My teeth feel wobbly when I eat. Is this worrying? I also have a feeling that from moving my teeth constantly I have risked them falling out at an early age