Eye possible problem? Should I worry?

I am 19 years old and when I was a kid I noticed that one eye would move while the other one stay still. I noticed that when I put my contacts in and out at times (so not frequently) my eye will move away while the other one doesn't. I normally look fine, my eyes respond well with eye movement, is only when I look to my left only (the problem does not occur if I look right) and focus (such as taking out a contact) that I notice this. I know I have ptosis and I have astigmatism.