Can a toothbrush cause dental composite bonding to wear?

Over
time, bonding could wear a little, but not that fast. Sometimes excess clear
bonding agent, the “glue” that holds the filling to the tooth, will break off
soon after an appointment. This is extra bonding agent that would be over an
area that wasn’t etched and losing it won’t hurt the actual filling. This would
probably look like a thin sliver of clear plastic. Brushing your teeth should
not cause any actual bonding to break off. It can’t hurt to have your dentist
take a look and make sure everything is fine if you’re not sure.

I would think that the bonding that came off was either excess bonding or bonding agent.  Have your dentist evaluate the area and determine if simple smoothing or replacement of the bonding is required.  Composite should not wear that fast and definitely does not occur in chunks.

Tooth brushing can cause wear on composite bonding, but it normally takes much longer to occur. It's possible that a small piece of the restoration could have flaked off. I would schedule an appointment with your dentist for an evaluation.