The tetragonal copper oxide Bi$_2$CuO$_4$ has an unusual crystal structure with a three-dimensional network of well separated CuO$_4$ plaquettes. This material was recently predicted to host electronic excitations with an unconventional spectrum and the spin structure of its magnetically ordered state appearing at T$_N$ $\sim$43 K remains controversial. Here we present the results of detailed studies of specific heat, magnetic and dielectric properties of Bi$_2$CuO$_4$ single crystals grown by the floating zone technique, combined with the polarized neutron scattering and high-resolution X-ray measurements. Our polarized neutron scattering data show Cu spins are parallel to the $ab$ plane. Below the onset of the long range antiferromagnetic ordering we observe an electric polarization induced by an applied magnetic field, which indicates inversion symmetry breaking by the ordered state of Cu spins. For the magnetic field applied perpendicular to the tetragonal axis, the spin-induced ferroelectricity is explained in terms of the linear magnetoelectric effect that occurs in a metastable magnetic state. A relatively small electric polarization induced by the field parallel to the tetragonal axis may indicate a more complex magnetic ordering in Bi$_2$CuO$_4$.
