ObjectivesRepeatability is crucial for ensuring the generalizability and clinical utility of radiomics-based prognostic models. This study aims to investigate the repeatability of radiomic feature (RF) and its impact on the cross-institutional generalizability of the prognostic model for predicting local recurrence-free survival (LRFS) and overall survival (OS) in esophageal squamous cell cancer (ESCC) receiving definitive (chemo) radiotherapy (dCRT).MethodsNine hundred and twelve patients from two hospitals were included as training and external validation sets, respectively. Image perturbations were applied to contrast-enhanced computed tomography to generate perturbed images. Six thousand five hundred ten RFs from different feature types, bin widths, and filters were extracted from the original and perturbed images separately to evaluate RF repeatability by intraclass correlation coefficient (ICC). The high-repeatable and low-repeatable RF groups grouped by the median ICC were further analyzed separately by feature selection and multivariate Cox proportional hazards regression model for predicting LRFS and OS.ResultsFirst-order statistical features were more repeatable than texture features (median ICC: 0.70 vs 0.42-0.62). RFs from LoG had better repeatability than that of wavelet (median ICC: 0.70-0.84 vs 0.14-0.64). Features with smaller bin widths had higher repeatability (median ICC of 8-128: 0.65-0.47). For both LRFS and OS, the performance of the models based on high- and low-repeatable RFs remained stable in the training set with similar C-index (LRFS: 0.65 vs 0.67, p = 0.958; OS: 0.64 vs 0.65, p = 0.651), while the performance of the model based on the low-repeatable group was significantly lower than that based on the high-repeatable group in the external validation set (LRFS: 0.61 vs 0.67, p = 0.013; OS: 0.56 vs 0.63, p = 0.013).ConclusionsApplying high-repeatable RFs in modeling could safeguard the cross-institutional generalizability of the prognostic model in ESCC.Critical relevance statementThe exploration of repeatable RFs in different diseases and different types of imaging is conducive to promoting the proper use of radiomics in clinical research.Key PointsThe repeatability of RFs impacts the generalizability of the radiomic model.The high-repeatable RFs safeguard the cross-institutional generalizability of the model.Smaller bin width helps improve the repeatability of RFs.