Abstract Background Triple Negative Breast cancer (TNBC) is incurable cancer with higher rates of relapse and shorter overall survival compared with other subtypes of breast cancer. Cellular retinoic acid binding protein 2 (CRABP2) belongs to fatty acid binding protein (FABP) family which binds with all-trans retinoic acid (RA). Previous studies from the database have reported the patients with high expression of CRABP2 showed different prognosis in ER+ and ER− breast cancer. However, its biological role and exact mechanism in breast cancer remain unknown. This aim of this study was to explore how CRABP2 regulated invasion and metastasis based on the estrogen receptor-α (herein called ER) status in breast cancer. Methods Immunohistochemical staining method was used to analyze the expression of CRABP2 in human breast cancer tissues. Lentivirus vector-based shRNA technique was used to test the functional relevance of CRABP2 knockdown in breast tumors. Tail vein injection model was used to examine the lung metastasis. Co-immunoprecipitation, Western blotting, immunofluorescence, and quantitative reverse transcription polymerase chain reaction (RT-qPCR) were conducted to investigate the underlying mechanism that influenced the ER to the regulation of CRABP2 to Lats1. Results We observed that knockdown of CRABP2 promotes EMT, invasion and metastasis of ER+ breast cancer cells in vitro and in vivo, whereas overexpression of CRABP2 yields the reverse results. In ER+ mammary cancer cells, the interaction of CRABP2 and Lats1 suppress the ubiquitination of Lats1 to activate Hippo pathway to inhibit the invasion and metastasis of ER+ mammary cancer. However, in ER− mammary cancer cells, the interaction of CRABP2 and Lats1 promote the ubiquitination of Lats1 to inactivate Hippo pathway to promote the invasion and metastasis of ER− mammary cancer. Conclusions Our findings indicate that CRABP2 can suppress invasion and metastasis of ER+ breast cancer and promote invasion and metastasis of ER− breast cancer by regulating the stability of Lats1 in vitro and in vivo, and it provides new ideas for breast cancer therapy.