Conservation laws in a quantum many-body system play a direct role in its dynamic behavior. Understanding the effect of weakly breaking a conservation law due to coherent and incoherent errors is thus crucial, e.g., in the realization of reliable quantum simulators. In this work, we perform exact numerics and time-dependent perturbation theory to study the dynamics of symmetry violation in quantum many-body systems with slight coherent (at strength $\lambda$) or incoherent (at strength $\gamma$) breaking of their local and global symmetries. We rigorously prove the symmetry violation to be a divergence measure in Hilbert space. Based on this, we show that symmetry breaking generically leads to a crossover in the divergence growth from diffusive behavior at onset times to ballistic or hyperballistic scaling at intermediate times, before diffusion dominates at long times. More precisely, we show that for local errors the leading coherent contribution to the symmetry violation cannot be of order lower than $ \propto \lambda t^2 $ while its leading-order incoherent counterpart is typically of order $\propto \gamma t$. This remarkable interplay between unitary and incoherent gauge-breaking scalings is also observed at higher orders in projectors onto symmetry (super)sectors. Due to its occurrence at short times, the diffusive-to-ballistic crossover is expected to be readily accessible in modern ultracold-atom and NISQ-device experiments.