Background Accurate size and stage estimation is important to monitor tumor response and plan further treatment in breast cancer patients undergoing neoadjuvant chemotherapy. To evaluate the accuracy of imaging findings [ultrasound (US) and magnetic resonance imaging (MRI)] for tumor size and stage estimations in early breast cancer patients and to elucidate the factors influencing tumor stage assessment. Methods We retrospectively enrolled consecutive women having pathologically confirmed breast cancer (stage T1/T2, 199 patients and 201 lesions) and preoperative records available for both US and MRI. The concordance between imaging-determined and pathological tumor size and stage was explored. The McNemar's test was conducted to compare the concordance between imaging-determined tumor size and imaging-determined tumor stage. Multivariate logistic regression was used to analyze the factors that influenced the accuracy. Results The concordance between US-determined and pathological tumor size (71.1%) was comparable to MRI-pathology concordance (72.6%). MRI-determined stage concordance (73.6%) was comparable to US-determined stage concordance (69.2%). Tumors with a larger pathological size, were more likely to be underestimated by US or MRI in terms of tumor size and stage (all P < 0.05). Conclusion Tumor size and tumor stage concordance did not significantly differ between US and MRI in early breast cancer patients; US could be the first choice for tumor size estimation and tumor staging.