The anisotropy of rare earth magnesium (Mg-RE) alloys has attracted significant attention. In this study, the room-temperature compression anisotropy of the extruded Mg-8.5Gd-4.5Y-0.8Zn-0.4Zr alloy was investigated utilizing techniques such as optical microscope (OM), electron backscatter diffraction (EBSD), and viscoplastic self-consistent (VPSC) modeling. The results show that the C0 (with the axis parallel to the extrusion direction) sample exhibited the greatest yield strength of 290 MPa, while the C45 (with the axis inclined at a 45° angle to the extrusion direction) and C90 (with the axis perpendicular to the extrusion direction) samples had yield strengths of 223 MPa and 250 MPa, respectively. The VPSC hardening parameters were significantly adjusted in this study based on the Schmid factor of deformation modes in Mg-RE alloy, particularly increasing the τ0 (Critical Resolved Shear Stress, CRSS) for basal slip. The ratios of CRSS for other deformation modes to basal slip were approximately as follows: CRSSTwin/CRSSBas = 2, CRSSPri/CRSSBas = 2.7 and CRSSPyr/CRSSBas = 3.3, while these ratios in non-rare earth magnesium alloys typically ranged between 0.8 and 3, 5–10 and 8–20. The stress-strain curves and pole figures obtained from the VPSC model exhibited a good agreement with experimental results. Based on the VPSC simulation results, the disparity in the activation level of basal slip was identified as the primary cause of mechanical anisotropy. Twinning only played a significant role in the early phases of deformation, which was not the most primary factor on yield anisotropy.