Contrast-enhanced ultrasound (CEUS), which provides more detailed microvascular information about the tumor, is always taken by radiologists in clinic diagnosis along with B-mode ultrasound (B-mode US). However, automatically analyzing breast CEUS is challenging due to the difference between the CEUS video and the natural video, e.g., sports or action videos, where the CEUS video has no positional displacements. Additionally, most existing methods rarely use the Time Intensity Curve (TIC) information of CEUS and non-imaging clinical (NIC) data. To address these issues, we propose a novel breast cancer diagnosis framework that learns the complementarity and correlation across hybrid modal data, including CEUS, B-mode US, and NIC data, by an adversarial adaptive fusion method. Furthermore, to fully exploit the CEUS information, the proposed method, inspired by the clinical processing of radiologists, first extracts the TIC parameters of CEUS. Then, we select a clip from CEUS using a frame screening strategy and finally get spatio-temporal features from these clips through a critical frame attention network. To our knowledge, this is the first AI system to use TIC parameters, NIC data, and ultrasound imaging in diagnoses. We have validated our method on a dataset collected from 554 patients. The experimental results demonstrate the excellent performance of the proposed method. The result shows that our method can achieve an accuracy of 87.73%, which is higher than that of uni-modal approaches by nearly 5%.Copyright © 2023 Elsevier Ltd. All rights reserved.