Recent years have seen strong progress in quantum simulation of gauge-theory dynamics using ultracold-atom experiments. A principal challenge in these efforts is the certification of gauge invariance, which has recently been realized in [B. Yang et al., arXiv:2003.08945]. One major but poorly investigated experimental source of gauge-invariance violation is an imperfect preparation of the initial state. Using the time-dependent density-matrix renormalization group, we analyze the robustness of gauge-invariant dynamics against potential preparation defects in the above ultracold-atom implementation of a $U(1)$ gauge theory. We find defects related to an erroneous initialization of matter fields to be innocuous, as the associated gauge-invariance violation remains strongly localized throughout the time evolution. A defect due to faulty initialization of the gauge field leads to a mild proliferation of the associated violation. Furthermore, we characterize the influence of immobile and mobile defects by monitoring the spread of entanglement entropy. Overall, our results indicate that the aforementioned experimental realization exhibits a high level of fidelity in the gauge invariance of its dynamics at all evolution times. Our work provides strong evidence that ultracold-atom setups can serve as an extremely reliable framework for the quantum simulation of gauge-theory dynamics.