Using spin-polarized scanning tunneling microscopy and density functional theory, we have studied the magnetic properties of Pd/Fe atomic bilayers on Re(0001). Two kinds of magnetic ground states are discovered due to different types of stacking of the Pd adlayer on Fe/Re(0001). For fcc-stacking of Pd on Fe/Re(0001), it is a spin spiral propagating along the close-packed direction with a period of about 0.9 nm, driven by frustrated exchange and Dzyaloshinskii-Moriya interactions. For the hcp stacking, the higher-order exchange interactions stabilize an uudd (up-up-down-down) state propagating perpendicular to the close-packed direction with a period of about 1.0 nm.
