Currently, there are intense experimental efforts to realize lattice gauge theories in quantum simulators. Except for specific models, however, practical quantum simulators can never be fine-tuned to perfect local gauge invariance. There is thus a strong need for a rigorous understanding of gauge-invariance violation and how to reliably protect against it. As we show through analytic and numerical evidence, in the presence of a gauge invariance-breaking term the gauge violation accumulates only perturbatively at short times before proliferating only at very long times. This proliferation can be suppressed up to infinite times by energetically penalizing processes that drive the dynamics away from the initial gauge-invariant sector. Our results provide a theoretical basis that highlights a surprising robustness of gauge-theory quantum simulators.