We study the moduli space $\mathcal{F}_{T_1}$ of quasi-trielliptic K3 surfaces of type I, whose general member is a smooth bidegree $(2,3)$-hypersurface of $\mathbb{P}^1\times \mathbb{P}^2$. Such moduli space plays an important role in the study of the Hassett-Keel-Looijenga program of the moduli space of degree $8$ quasi-polarized K3 surfaces. In this paper, we consider several natural compactifications of $\mathcal{F}_{T_1}$, such as the GIT compactification and arithmetic compactifications. We give a complete analysis of GIT stability of $(2,3)$-hypersurfaces and provide a concrete description of the boundary of the GIT compactification. For the Baily--Borel compactification of the quasi-trielliptic K3 surfaces, we also compute the configurations of the boundary by classifying certain lattice embeddings. As an application, we show that $(\mathbb{P}^1\times \mathbb{P}^2,\epsilon S)$ with small $\epsilon$ is K-stable if $S$ is a K3 surface with at worst ADE singularities. This gives a concrete description of the boundary of the K-stability compactification via the identification of the GIT stability and the K-stability. We also discuss the connection between the GIT, Baily--Borel compactification, and Looijenga's compactifications by studying the projective models of quasi-trielliptic K3 surfaces.
