Abstract Quantum annealing, which involves quantum tunnelling among possible solutions, has state-of-the-art applications not only in quickly finding the lowest-energy configuration of a complex system, but also in quantum computing. Here we report a single-crystal study of the frustrated magnet α-CoV2O6, consisting of a triangular arrangement of ferromagnetic Ising spin chains without evident structural disorder. We observe quantum annealing phenomena resulting from time-reversal symmetry breaking in a tiny transverse field. Below ~ 1 K, the system exhibits no indication of approaching the lowest-energy state for at least 15 hours in zero transverse field, but quickly converges towards that configuration with a nearly temperature-independent relaxation time of ~ 10 seconds in a transverse field of ~ 3.5 mK. Our many-body simulations show qualitative agreement with the experimental results, and suggest that a tiny transverse field can profoundly enhance quantum spin fluctuations, triggering rapid quantum annealing process from topological metastable Kosterlitz-Thouless phases, at low temperatures.