The recommendation is to wear the aligners for 22 hours a day. This allows approximately 2 hours for eating, brushing, flossing, and drinking during the day.
Teeth move when there is a constant pressure placed on them. There is some resorption of the bone on the pressure side, and formation of bone as the tooth moves. Unfortunately, the teeth aren't smart enough to know whether it is plastic or braces moving them. All they know is that there is some pressure on them.
Since conventional braces are attached to your teeth, you really don't have much of a choice about keeping the pressure, albeit light, on your teeth. With aligners, it really is your choice to wear them or not, but to keep treatment on time and on track, wear them as close to full-time as possible.