The majority of the iron-based superconductors (FeSCs) exhibit a two-dimensional square lattice structure. Recent reports of pressure-induced superconductivity in the spin-ladder system, BaFe$_2$X$_3$ (X =S,Se), introduce a quasi-one-dimensional prototype and an insulating parent compound to the FeSCs. Here we report X-ray, neutron diffraction and muon spin relaxation experiments on BaFe$_2$Se$_3$ under hydrostatic pressure to investigate its magnetic and structural properties across the pressure-temperature phase diagram. A structural phase transition was identified at a pressure of 3.7(3) GPa. Neutron diffraction measurements at 6.8(3) GPa and 120 K show that the block magnetism persists even at these high pressures. A steady increase and then fast drop of the magnetic transition temperature $T\rm_N$ and greatly reduced moment above the pressure $P_s$ indicate potentially rich and competing phases close to the superconducting phase in this ladder system.
