My orthodontist told me to only wear my retainer at night, should I be worried about teeth shifting?

Yesterday, I got my braces off after 2.5 years. My orthodontist bonded a permanent retainer to my bottom 6 teeth and top 4 teeth. He then gave me a clear Essix retainer for the top teeth only and said to wear it during the night. Since I don't have a bottom retainer, should I be worried about my bottom teeth shifting? Also is only night time wear enough to keep my top teeth the way they are now?