Background Deep learning (DL) models, particularly those based on Transformer architecture, which are capable of capturing complex patterns and dependencies in medical imaging data, have shown great potential in improving brain tumor prognosis and guiding treatment decisions. However, the effectiveness of Transformer-based models, especially in predicting recurrence after treatment, has yet to be fully demonstrated.Purpose This study aims to develop and validate a Transformer-based DL model that utilizes multi-modal data, specifically pre-treatment magnetic resonance imaging (MRI) scans fused with radiotherapy dose (RTDose) information, to predict post-treatment recurrence in brain tumors, thereby providing decision support for personalized radiotherapy.Methods In this study, we employed MRI data from patients with brain metastases who had undergone Gamma Knife radiosurgery at the University of Mississippi Medical Center to train and validate a Transformer-based DL model. To further validate the Transformer-based model, a comparative analysis was conducted with nine established prognostic models. The generalizability and predictive accuracy of the model were validated across multiple clinical subgroups. To further exclude other potential factors influencing brain tumor recurrence, logistic regression (LR) and statistical analysis were conducted to confirm the independence of the model's predictions.Results The model achieved an average area under the receiver operating characteristic curve (AUROC) of 0.817 on 3-fold cross-validation, outperforming all other models. The model also exhibited strong generalizability across clinical subgroups, with AUROCs of 0.806 for patients under 50, 0.723 for those aged 51-60, and 0.843 for those aged 61-77 (p=0.057$p = 0.057$). For gender subgroups, the AUROCs were 0.783 for females and 0.820 for males (p=0.057$p = 0.057$). LR analysis confirmed the independence of the model's predictions, with a largest permutation importance and p<0.001$p < 0.001$.Conclusions The Transformer-based DL model developed in this study serves as a reliable prognostic tool for predicting brain tumor recurrence following radiotherapy. It demonstrated superior performance compared to nine established prognostic models, including various deep learning architectures and radiomics-based methods, and holds the potential to guide personalized treatment strategies for brain tumor patients.