What happens if I just had a root canal and the file broke by the root of my molar?

Unfortunately there is no definite answer for you. Much of the possibility of what happens to your tooth depends on when during the process the file separated. If the tooth had been cleaned out well prior to breaking, the odds are better that your tooth will be fine long term. If it broke early in the process, your tooth may not have been cleaned out and the tooth can become infected. I've seen  teeth with broken files last indefinitely, so I would certainly not worry about it since there isn't anything you can do to change anything and you might be ok long term. I hope this helps!

The dentist did the correct thing to refer you to the specialist without delay. If the smallest part of the file has broken in the canal then as my colleague has suggested, things many settle almost permanently without any further problems. 
If the amount that has been left behind in the canal is the area that there is infection, then there is the possibility that an apicoectomy can be done at a later stage to save the tooth by removing both the broken file fragment and to clean the apical area of bacterial contaminants and infection.
Finally, if the tooth is not part of a bridge or other significant structure in your mouth, replacing it with an implant when the time arises could be a solution. The important things is to keep your denitst informed so that treatment can be organised in every instance without decay. 
Hope this further expands on and lets you relax a little about what has happened in your tooth.  
kind regards
Thomas Dey, BS, MSc