Invisalign don't track

I can understand why you would be concerned, but fear not, Invisalign is a very safe method of aligning teeth, and your dentist has likely encountered other patients who required refinements during treatment as most of us have. Moving teeth can be complicated and at times unpredictable. As advanced as Invisalign and its treatment simulations (ClinCheck software) are, they still can't perfectly predict the future. Also, there is variability between patients which can be due to a wide variety of reasons.
Your main question was whether it would be harmful to continue to change Invisalign trays, and the short answer is no, it is extremely unlikely to cause harm to your teeth. However, it may become difficult, if not impossible to change your trays if the teeth get too far behind the trays. So, it is important that you and your dentist come up with a plan to revise the treatment plan quickly.