Background: Esophageal squamous cell cancer (ESCC) is a disease with a male predominance. Accordingly, the applicability of prognostic indicators values previously set for the general population with ESCC has not been reported for determining the physical state in females. Methods: Patients with ESCC were pooled from 2009 to 2017 at Sichuan Cancer Hospital. We determined the differences in the nutritional and inflammatory indicators between gender by sex-stratified survival analysis in all cohorts (n = 2,660) and matching cohorts (n = 483 pairs) separately. Propensity score matching (PSM) was employed to eliminate selection bias between genders. We further performed the prognostic value of total cholesterol (TC) by subgroup analysis in the female cohort. The area ROC curve was used to assess the predictive performance of TC in females. Results: There were a total of 2,660 patients with ESCC, of whom 2,173 (81.7%) were male and 487 (18.3%) were female. Before PSM, the prognostic nutritional index was an independent factor for OS in males but not in females. For cohort with or without matching, TC was an independent prognostic factor in females not for males. Furthermore, female patients with high TC level had significant poor OS in stages III and IV. The AUCs of TC were 0.63 and 0.70 for predicting 3- and 5-year OS, respectively. Conclusion: Based on a much larger cohort, we confirmed that gender was a significant prognostic factor for ESCC patients. Interestingly, we found a significant difference in TC related to ESCC prognosis between genders. Collectively, TC might be an independent prognostic factor in females with ESCC.