Background Changes in Polyamine metabolism (PAM) have been shown to establish a suppressive tumor microenvironment (TME) and substantially influence the progression of cancer in the recent studies. However, newly emerging data have still been unable to fully illuminate the specific effects of PAM in human cancers. Here, we analyzed the expression profiles and clinical relevance of PAM genes in colorectal cancer (CRC). Methods Based on unsupervised consensus clustering and principal component analysis (PCA) algorithm, we designed a scoring model to evaluate the prognosis of CRC patients and characterize the TME immune profiles, with related independent immunohistochemical validation cohort. Through comparative profiling of cell communities defined by single cell sequencing data, we identified the distinct characteristics of polyamine metabolism in the TME of CRC. Results Three PAM patterns with distinct prognosis and TME features were recognized from 1224 CRC samples. Moreover, CRC patients could be divided into high- and low-PAMscore subgroups by PCA-based scoring system. High PAMscore subgroup were associated to more advanced stage, higher infiltration level of immunosuppressive cells, and unfavorable prognosis. These results were also validated in CRC samples from other public CRC datasets and our own cohort, which suggested PAM genes were ideal biomarkers for predicting CRC prognosis. Notably, PAMscore also corelated with microsatellite instability-high (MSI-H) status, higher tumor mutational burden (TMB), and increased immune checkpoint gene expression, implying a potential role of PAM genes in regulating response to immunotherapy. To further confirm above results, we demonstrated a high-resolution landscape of TME and cell–cell communication network in different PAM patterns using single cell sequencing data and found that polyamine metabolism affected the communication between cancer cells and several immune cells such as T cells, B cells and myeloid cells. Conclusion In total, our findings highlighted the significance of polyamine metabolism in shaping the TME and predicting the prognosis of CRC patients, providing novel strategies for immunotherapy and the targeting polyamine metabolites.