Is this swelling after removal permanent or will it subside?

It is not unusual for swelling to last a long time when the dissection is in the sub-periosteal plane. This is under the lining of the bone and where implants are generally placed. I would expect the shape of your chin to be somewhat more projected than it was prior to the surgery because of the capsule that forms around the implant. How much bigger is hard to say. However, most patients who undergo an explantation are usually satisfied with the end result even if the chin doesn't go back exactly to where it was originally.

Chin Implant removal information/and considerations: For the vast majority of chin implants, removal is straight  forward most commonly using the  same incision that was used to place it. The procedure is often done in the office with local anesthesia with or without some sedation, but also commonly done in the OR under IV sedation or general anesthesia per patient's and surgeon's preference.  Most implants are silicone and are not adherent so slip out readily. Medpor, also called PTFE, may have tissue ingrowth and can be a bit more involved but still can be removed in the same manner.  It is very helpful for your surgeon to  review your prior surgical records to know the size and type of implant used. This may be obtained by contacting your prior surgeon's office or OR facility and request your prior surgical records.
Expectations: if the chin implant was small and removed within the first few years chances are likely that you will return a similar size and shape as before. If the implant was small, the chin soft tissue usually shrinks back down without  sagging.  Often, there is still some gain once the implant is removed due to the scarring and capsule (lining your body makes around the implant) that gives some benefit. On the other hand a large implant placed many years ago, and other factors such as being elderly, having osteoporosis and perhaps smoking that can cause loss of underlying bone due to  bony erosion from the pressure of the implant. In this case  use of a natural filler such your fat or at times a bone graft may recommended for different reasons. Your surgeon may request a  Panorex or lateral cephalometric x-ray to see if this is the case.  Temporary fillers can also be tried such as Jeuvederm, Sculptra, Voluma, Volbella, etc. 
Most important of all make sure you pick a plastic surgeon with great expertise in #chinaugmentation and #chinimplantremoval. 
See the below link on how best to choose your surgeon.

Generally speaking, it can take 6 to 12 months to completely heal after surgery. Please keep in touch with your surgeon or his/her clinical team to discuss your ongoing concerns.