Background The effects of childhood adversities on cognitive function in later life are well reported. However, few studies have examined the cumulative mechanism, especially in Chinese population. This study aims to explore this cumulative effects of childhood adversities on mid to late cognitive decline in China. Methods Data were drawn from the second and third wave of the China Health and Retirement Longitudinal Study (CHARLS). We included 9,942 respondents aged 45 and above and retrospectively collected information on childhood adversities. Cognitive function was measured in three dimensions: orientation and calculation, immediate memory, and delayed memory. A structural equation model was employed for analysis. Results Age (beta = -0.155, P<0.001) and mid to late depressive symptoms (beta = -0.041, P<0.001) showed direct effects on cognitive decline. Low mid to late life socioeconomic status (SES) showed a direct effect on mid-late cognitive impairment (beta = 0.603, P<0.001) and an indirect effect through depression (beta = 0.007, P<0.001). Low childhood SES (beta = 0.310, P<0.001), lack of friends (beta = 0.208, P<0.001), parental mental health problems (beta = 0.008, P<0.001), and poor relationship with parents (beta = 0.001, P<0.001) had an indirect effect on cognitive impairment. Conclusions Childhood adversities had negative effects on cognitive function among middle aged and elderly population in China. The findings suggest that early counter measures on childhood adversities may lead to an effective reduction of cognitive impairment.