Vortices are topological defects of type-II superconductors in an external magnetic field. In a similar fashion to a quantum anomalous Hall insulator, quantum anomalous vortex (QAV) spontaneously nucleates due to orbital-and-spin exchange interaction between vortex core states and magnetic impurity moment, breaking time-reversal symmetry (TRS) of the vortex without an external field. Here, we used scanning superconducting quantum interference device microscopy (sSQUID) to search for its signatures in iron-chalcogenide superconductor Fe(Se,Te). Under zero magnetic field, we found a stochastic distribution of isolated anomalous vortices and antivortices with flux quanta $\Phi_0$. By applying a small local magnetic field under the coil of the nano-SQUID device, we observed hysteretic flipping of the vortices reminiscent of the switching of ferromagnetic domains, suggesting locally broken-TRS. We further showed vectorial rotation of a flux line linking a paired vortex-antivortex with the local field. These unique properties of the anomalous vortices satisfied the defining criteria of QAV. Our observation suggests a quantum vortex phase with spontaneous broken-TRS in a high-temperature superconductor.