Rectal neuroendocrine neoplasms (rNENs) are among the most frequent gastrointestinal neuroendocrine neoplasms and pose a serious challenge for clinical management. The size of the primary neoplasm is considered to be the most important predictor of disease progression, but the genetic alterations that occur during the progression of rNENs remain unknown. Here, we performed a comprehensive whole-exome sequencing study on 54 tumor-normal paired, formalin-fixed paraffin-embedded specimens from patients locally diagnosed with rNENs. Of these, 81.5% (n = 44) were classified as small-sized (<= 2 cm) rNENs, while the remainder (18.5%, n = 10) were classified as large-sized (>2 cm) rNEN samples. Comparative analysis revealed marked disparities in the mutational landscape between small- and large-sized rNEN samples, and between large-sized rNEN samples with or without lymph node metastases. The high-confidence driver genes RHPN2, MUC16, and MUC4 were significantly mutated in both small- and large-sized rNEN specimens, whereas mutations in MAN2A1, and BAG2 were only identified in large-sized specimens diagnosed with lymph node metastases. Correspondingly, we observed that the mTOR and MAPK pathways were preferentially enriched in the large-sized rNEN specimens. Signature-based analysis revealed that mutational processes associated with defective DNA base excision repair (SBS30) significantly accumulated in large-sized rNEN samples with lymph node metastases, highlighting the important role of this mutagenic process in promoting rNEN progression. We further found that most rNEN subjects, regardless of tumor size, harbored at least one alteration with targeted therapeutic implications. Taken together, these results elucidate the genetic features associated with tumor size and lymphatic metastasis in rNEN patients, which will deepen our understanding of the genetic changes during rNEN progression and potentially directing improvements in rNEN treatment strategies.