The structure of dielectric perovskite BaZrO$_3$, long known to be cubic at room temperature without any structural phase transition with variation of temperature, has been recently disputed to have different ground state structures with lower symmetries involving octahedra rotation. The pressure-dependent Raman scattering measurements can identify the hierarchy of energetically-adjacent polymorphs, helping in turn understand its ground state structure at atmospheric pressure. Here, Raman scattering spectra of high-quality BaZrO$_3$ single crystals grown by the optical floating zone method are investigated in a pressure range from 1 atm to 42 GPa. First, based on the analyses of the infrared and Raman spectra measured at the atmosphere, it is found that all observed vibrational modes can be assigned according to the cubic $Pm\bar{3}m$ structure. In addition, by applying pressure, two structural phase transitions are found at 8.4 and 19.2 GPa, one from the cubic to the rhombohedral $R\bar{3}c$ phase and the other from the rhombohedral to the tetragonal $I4/mcm$ phase. Based on the two pressure-induced structural phase transitions, the true ground state structure of BaZrO$_3$ at room temperature and ambient pressure is corroborated to be cubic while the rhombohedral phase is the closest second.