Breast adipose tissue (AT) participates in the physiological evolution and remodeling of the mammary gland due to its high plasticity. It is also a favorable microenvironment for breast cancer progression. However, information on the properties of human breast adipose progenitor cells (APCs) involved in breast physiology or pathology is scant. We performed differential enzymatic dissociation of human breast AT lobules. We isolated and characterized two populations of APCs. Here we report that these distinct breast APC populations selectively expressed markers suitable for characterization. The population preferentially expressing ALPL (MSCA1) showed higher adipogenic potential. The population expressing higher levels of INHBA and CD142 acquired myofibroblast characteristics upon TGF-β treatment and a myo-cancer-associated fibroblast profile in the presence of breast cancer cells. This population expressed the immune checkpoint CD274 (PD-L1) and facilitated the expansion of breast cancer mammospheres compared with the adipogenic population. Indeed, the breast, as with other fat depots, contains distinct types of APCs with differences in their ability to specialize. This indicates that they were differentially involved in breast remodeling. Their interactions with breast cancer cells revealed differences in the potential for tumor dissemination and estrogen receptor expression, and these differences might be relevant to improve therapies targeting the tumor microenvironment.