The purpose of this study is to determine the best dose processing method for deep learning-based dose prediction in brachytherapy (BT), as well as to investigate the feasibility of using the inverse dose optimization algorithm to improve treatment planning quality.BT data from 186 patients with cervical cancer were retrospectively collected. The data were divided into three sets: training, validation, and test, with a ratio of 150:18:18. The dose data was normalized using square-root transformation normalization, logarithmic normalization, and linear normalization. For dose distribution prediction, a 3D U-Net architecture was used. The predicted results were compared to unprocessed dose data. The four groups of dose predictions were assessed using the Dice similarity coefficient (DSC), conformity index (CI), and homogeneity index (HI). The group with the best overall performance was chosen, and the dose prediction results were fed into a gradient-based planning optimization (GBPO) algorithm for additional optimization. The target D90 % was normalized to 6 Gy. The D1cc and D2cc of the OARs were compared prior to and following optimization.The dose prediction method using unprocessed doses produced the best overall performance on the DSC, CI, and HI metrics. The (DSC, CI, HI) values for unprocessed dose, square-root transformation normalized, log normalized, and linear normalized were (0.94, 0.74, 0.49), (0.93, 0.72, 0.50), (0.91, 0.71, 0.45) and (0.90, 0.71, 0.47), respectively. The predicted dose results for the unprocessed dose group were further optimized by the GBPO algorithm. The outcomes demonstrated that the (D1cc, D2cc) values for the bladder, rectum, and sigmoid decreased by (2.11 %, 2.09 %), (2.62 %, 2.14 %) and (3.16 %, 2.98 %), respectively, and were statistically significant (p < 0.05). The small intestine dose increased slightly; the average increase in the D1cc and D2cc doses was 2.08 % and 1.63 %, respectively, with no statistically significant difference (P > 0.05).When using deep learning for BT dose prediction in the 3D U-Net model with the cervical cancer BT data used in this study, dose normalization processing is not recommended The predicted dose can be further optimized using inverse dose optimization algorithms to improve the treatment plan's quality.Copyright © 2025 Elsevier Ltd. All rights reserved.