Can a varicocele harm pubertal development.

Great Question. Varicoceles may have an impact on the testicle that the varicocele is surrounding, causing it to be smaller. The  vast majority occur  on the left side.  Because of the excess blood, the temperature of the testicle on the affected side may be hotter, leading to decreased sperm production. 
That said, it would only affect fertility, but not development. It is possible that the signs and symptoms you describe [no facial hair, shorter penis, delated puberty] may be associated with the same underlying cause that contributed to the development of the varicocele. If you are concerned, seek the opinion of a doctor who is experienced with puberty and pubertal changes, they may order some blood and genetic tests to ensure that everything is normal and you are just a "late bloomer"...
Hope this helps.