What will happen if I remove chin implant after 3 years?

Chin Implant removal information/and considerations: For the vast majority of chin implants, removal is straight  forward most commonly using the  same incision that was used to place it. The procedure is often done in the office with local anesthesia with or without some sedation, but also commonly done in the OR under IV sedation or general anesthesia per patient's and surgeon's preference.  Most implants are silicone and are not adherent so slip out readily. Medpor, also called PTFE, may have tissue ingrowth and can be a bit more involved but still can be removed in the same manner.  It is very helpful for your surgeon to  review your prior surgical records to know the size and type of implant used. This may be obtained by contacting your prior surgeon's office or OR facility and request your prior surgical records.
Expectations: if the chin implant was small and removed within the first few years chances are likely that you will return a similar size and shape as before. If the implant was small, the chin soft tissue usually shrinks back down without  sagging.  Often, there is still some gain once the implant is removed due to the scarring and capsule (lining your body makes around the implant) that gives some benefit. On the other hand a large implant placed many years ago, and other factors such as being elderly, having osteoporosis and perhaps smoking that can cause loss of underlying bone due to  bony erosion from the pressure of the implant. In this case  use of a natural filler such your fat or at times a bone graft may recommended for different reasons. Your surgeon may request a  Panorex or lateral cephalometric x-ray to see if this is the case.  Temporary fillers can also be tried such as Jeuvederm, Sculptra, Voluma, Volbella, etc. 

Most important of all make sure you pick a plastic surgeon with great expertise in #chinaugmentation and #chinimplantremoval. 

You could either place a smaller implant or go for a sliding genioplasty, which is done in office and take 1 hour to do. It also gives a natural look.

Lets evaluate your concerns to transform your result to a positive experience and optimal cosmetic outcome. 
We can start by acquiring your operative report from your surgery. This report is dictated by the surgeon and reviews each step and observations noted. Better yet, return to the surgeon who placed the implant and have them review all your concerns. If you feel uncomfortable with that recommendation, find a competent facial implant surgeon (facial plastic or plastic surgeon with facial plastic surgery experience).
First: chin numbness. 
Examination to determine if one or both sensory nerves (mental nerve) that come out of the bone between your 1st and 2nd premolar teeth about half way down the height of your jaw bone (mandible). An Xray may assist in determining the present estimated position of the implant or a CT scan  can more accurately reveal the position of the implant to estimate if the implant is resting on the mental sensory nerve. 
If the numbness was noted immediately after surgery, the sensory nerve was most likely injured during the procedure. If it is not imp;roved by 18 months, it will most likely not improve. The only exception, in some patients where the implant is resting on the nerve and then removed, the sensation has returned.
Second: manibile pain and possible intermittent swelling
The implant resting one the sensory nerve may cause pain. The other consideration is an infection. Dental radiographs by your dentist can rule out a periodical (tooth) infection or abscess. Physical examination associated with a type of radiograph will assist in identifying the problem. If this preoperative examination does not yield a diagnosis, then during the revision surgery, examination for the cause of numbness and pain is sought.
Third:  Goal: tightening the jawline.
The chin implant should be able to be salvaged and determining the correct size is imperative. Either leaving the implant in or removing it, there may be an issue with the jawline.
Removing the implant may result in persistent volume loss at the chin and pre-jowl sulcus. Facial fat transfer or dermal fillers such as Bellafill or Radiesse may be used. Alternatively, an osteotomy (cutting the bone) and advancing the bone (anterior genioplasty) can be used to enhance the chin.
Skin tightening can be performed externally with skin tightening procedures such as RF or radio frequency. Minimally invasive procedures can use RF (Thermi) or ultrasound (VASER) to tighten the skin from under the skin. 
Finally, a facelift can suspend drooping or ptosis soft tissue. 
As you can tell from y response, your situation is more complex than those of some patients. Finding a surgeon you are comfortable with is the first step in moving forward. 
In summary, your situation can be remedied to provide a satisfactory cosmetic result and I hope resolution of your numbness and chin swelling and discomfort.