Thirteen previously undescribed guaiane-type sesquiterpenoids based on [5,7] bicyclic system, stelleranoids A–M (1–13), along with six known analogues (14–20), were isolated from the roots of Stellera chamaejasme with chromatographic techniques. Their structures including absolute configurations were determined by HRESIMS and spectroscopic data, quantum chemical calculations, as well as X-ray crystallographic analysis. Cytotoxicity test in three cell lines indicated that compound 14 had relatively stronger cytotoxic effect against MKN-45, SKOV3, and Du145 cell lines with IC50 of 9.8, 17.4 and 7.3 μM, respectively; compounds 3 and 8 displayed moderate cytotoxic effect against MKN-45 and Du145 cell lines with IC50 ranged from 14.5 to 18.8 μM, comparable to those of the positive control. As determined by fluorescent microscopy and flow cytometry in Du145 cell line, compound 14 could promote cell apoptosis and cause cell cycle arrest at the G0/G1 phase, leading to the inhibition of cell proliferation. Further Western blot analysis revealed that this inhibitory effect was accompanied by upregulating pro-apoptosis proteins cleaved-PARP, cleaved-Caspase-9 and tumor suppressor protein p53 while downregulating anti-apoptotic protein Bcl-2 in 14-treated Du145 cells.