Is It Safe to Use a Scrub Towel on Your Face?

I lather my face in the shower with facial cleanser and then wet a scrub towel and apply a bit of cleanser on the towel as well. I then apply the towel to my face in circular motions very lightly, applying almost no pressure. Some people still tell me this is too abrasive for facial skin. Is this true? I have been using this method for a long time but I cannot tell if it is harming my skin because I am only 23.