Can I wear my retainer too much? Do I need to leave it out often for my teeth to settle?

I finished invisaline treatment about 6 months ago. After a month with a retainer I was told that I could move to wearing the retainer only at night, but I have to wear them fairly often because my teeth just keep shifting and aching. I asked my orthodontist this past week and he said I'd been wearing my retainers too much. I was originally told there was no such thing as wearing them too much by employees in his office and I can't find any support for what he's said.