Background: The low computation efficiency impeded the broad application of Monte Carlo (MC) simulation to particle therapy. The existing deep learning (DL) methods for fast dose calculation lacked physics-based interpretability, hence, may introduce additional risks, especially for the more complex carbon ion radiotherapy. Purpose: To develop and validate a multi-modal diffusion model, Diff-MC, for noise reduction of particle number limited MC dose calculation, potentially supporting better optimization and faster online adaptation for carbon ion radiotherapy. Methods: By using multi-modal data such as CT images, dose maps using a low number of primary particles and beam parameters, and so forth, Diff-MC was developed to generate a dose map adaptively based on the beam state. To enable effective inter-modal interactions, a hybrid-fusion strategy was applied to integrate the data-level, feature-level, and decision-level fusion. The model was evaluated on a highly heterogeneous dataset, including 15 000 paired beamlet data cropped from 20 CTs for training and validating, 500 paired beamlet data cropped from other 5 CTs for testing, and 500 paired beamlet data cropped from another 100 CTs for generalizability test. All datasets encompassed various geometry and beamlet physics parameters such as energy distribution and number of primary particles, and so forth. Results: Using the MC simulation based on high number of primary particles as ground-truth, the Diff-MC achieved nearly linear acceleration and high accuracy of gamma passing rate up to 99.25% under the criteria of 3 mm, 3%, 10% cutoff. The performance was significantly higher (all ) than the UNet-based models (96.17%) and transformer-based models (97.81%). The accuracy achieved by Diff-MC in the generalizability test was 99.22%. The lateral dose, integral depth dose (IDD), and percentage depth dose (PDD) of Diff-MC were also more consistent with the ground-truth than that of conventional AI models. Conclusions: The proposed Diff-MC method displayed high efficiency and robustness in carbon ion dose calculation. By maintaining the physics features of MC, the results of Diff-MC were more interpretable and generalizable than the conventional AI models.