Abstract Background Immune cells and stromal cells in the tumor microenvironment play a vital role in the progression of colorectal cancer (CRC). The study aimed to screen valuable prognostic biomarkers in CRC based on stromal and immune scores. Method The ESTIMATE algorithm was used to calculate the immune and stromal scores of CRC samples in TCGA. Then samples were divided into high and low score groups based on the median value of the scores. Differentially expressed genes (DEGs) associated with immune and stromal scores were screened. WGCNA and univariate COX regression analysis were performed to further identify key prognostic genes. Analysis of scRNA-seq for CRC was used for verifying the main source of the key genes. The prognostic value of they was validated based on The Gene Expression Profiling Interactive Analysis and GSE17536 dataset. TIMER and CIBERSORT algorithms were applied to analyze the correlations among key genes and tumor-infiltrating immune cells. Several pairs of colon cancer tissue were used to be proven. Result 1314 upregulated and 4 downregulated genes were identified, which were significantly enriched in immune-related biological processes and pathways. Among these DEGs, SPOCK1 and POSTN were identified as key prognostic genes and mainly expressed in cancer-associated fibroblasts for CRC. High expression of SPCOK1 and POSTN was associated with advanced clinical stage, T stage, N stage, and poor prognosis of CRC. The results from CIBERSORT and TIMER revealed that SPOCK1 and POSTN were associated with tumor-infiltrating immune cells, especially macrophages and neutrophils. Meanwhile, in several pairs of human colorectal tissue samples, SPOK1 and POSTN were found to be significantly overexpressed in colorectal tissue compared with para-cancer tissue, and macrophage surface markers CD68 (co-expressed by M1 and M2 macrophages) and CD206 (M2-specific macrophage expression) were also overexpressed in cancer tissue. Besides, SPOCK1 and POSTN expression were positively correlated with the expression of immune checkpoints. Conclusion Collectively, our results indicate that SPOCK1 and POSTN associated with CAF may be novel prognostic biomarkers in CRC and correlate with immune infiltrates.