Retainers just make my gap become larger and visible?

When i took my braces off, my teeth are perfectly align. But there is "small line" between my left lateral and central incisors, meaning that these two teeth are not touching each other completely if u see closely. I stop wearing my retainer for 5days and i notice my left lateral incisors shifted slightly to the back making the "small line" gap become less visible. I try wearing my retainer back for overnight, my left lateral incisors was back to the alignment but it leave a gap as before.